energy meters: Linux RAPL and NVML on real hardware; an NVML counter past its board is Implausible #129
Loading…
Reference in a new issue
No description provided.
Delete branch "energy/meters-on-real-linux-hardware"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The ten commits sit directly on
main, which now holds #110, #114–#117 and #120–#124. Before this they sat on a merge of those branches, and the forge couldn't compute a clean merge base for that history. The tree is byte-identical tomainmerged with the earlier head, and that tree was compiled in all 9 dependent workspaces and tested in jcp-rs (369), eoc-rs (717), sandbox-rs, joule-ui-rs (72) and proof-rs (139).Until now, the Linux RAPL and NVML paths in all five energy meters had only been tested against fixture sysfs trees. This PR runs them on real hardware. jetson-hub has a 13th-gen Intel i9 with powercap zones
package-0,core,psysand an mmio package, plus an RTX 4050 Laptop GPU (driver 595.84). Each test checks the meter against an oracle the test reads for itself.RAPL: correct everywhere
Each crate gets a
tests/rapl_hardware.rs: eoc-meter, sandbox-meter, sandbox-exec-microvm, jcp-energy, jouleclaw-energy and proof-cascade. Each test asserts:package-Nzones read just before and just after);HwShunt(orMeasured) only when live.Example from jcp-energy: the meter read 7,664,287 µJ against a bracket of 7,647,990–7,664,287 µJ. In the same window psys moved 61 mJ and the core sub-zone 6.35 J; neither was added.
Mutations caught: psys summed as a package, the core sub-zone summed, the mmio zone summed, and
energy_ujread as mJ. One limit remains: under a package at its power limit (load 11), proof's busy-minus-idle figure can read 0 and miss a core-zone double count.NVML: was wrong in all four crates, fixed
The RTX 4050 Laptop GPU's total-energy counter advances steadily, so every crate called it live and graded it as hardware. It is not the board's energy:
Independent check, reading NVML directly through
ctypes:Rule (sandbox-meter, jcp-energy, jouleclaw-energy, eoc-meter): each probe reads the board's power draw at every sample and its maximum power limit once. A counter that advances is
Implausible, and never used, if any of these holds:jouleclaw then falls back to its power-draw integral, graded
ModelBased. The others decline the rung. Each crate has atests/nvml_hardware.rsthat failed on jetson-hub before its fix (crate said Live, oracle said Implausible) and passes after.eoc-meter:
detect()ranked NVML ahead of RAPL. eoc grades any advancing counterMeasured, and it has no coverage field. So a GPU-only figure became the whole cascade's measurement. NVML is no longer adetect()rung, andNvmlCounteris no longer aJouleCounter: it is read withread_gpu_microjoules(), so no consumer can grade it as the cascade's counter.Also:
Verified
On jetson-hub: every crate's lib tests plus its hardware tests, with NVML features on. I re-ran jcp-energy independently from this branch's tree: RAPL Live /
HwShunt/Wholeinside the bracket, NVML Implausible (75.9 W counter against a 13.3 W board), 32 lib tests.On macOS:
--features nvml)--features cuda)eoc's workspace check passes as well.
In CI: Docker masks
/sys/devices/virtual/powercap(default masked paths, unprivileged), and the container has no GPU. So in CI these tests take their "unavailable, and the oracle agrees" branch. The live branch runs on a host that exposes the counters, such as jetson-hub itself.🤖 Generated with Claude Code
This crate's contract, in its own module doc: "measured when the platform exposes it, estimated when it doesn't, and the provenance always says which ... A reading never silently upgrades its own provenance." The IOReport rung broke that on current macOS, and the CI job meant to catch it could not. What IOReport actually returns on macOS 27.0.1, measured by dumping the channels rather than re-running the test: - On an M5 Max, every CPU rail in the Energy Model group (`CPU Energy`, `PCPU`, `MCPU*`, `PACC_*`) holds the same cumulative value across 200 ms, 1 s, 3 s and ~4 minutes of full CPU load. `CPU Energy` read 73,176,481 mJ before and after. `GPU Energy` stays live. - On an M3 Ultra, the same OS: all 645 Energy Model rails are frozen, CPU, DRAM and display included. - Every channel is IOReport format 1 (Simple), so the accessor is right, and subscribing to the Energy Model group alone instead of all channels does not unfreeze anything. No channel in any group exposes live CPU energy to an unprivileged reader; "CPU Stats" residency moves, but that is activity. So `detect()` selected IOReport the moment a subscription opened, and `measure()` accepted any `a > b` on the summed rails as the task's energy. On the M5 Max the GPU rail makes `a > b` true: GPU energy was reported as the energy of CPU work, as a whole-task ModelBased figure. On the M3 Ultra nothing moves, so it fell to time×power under the same tag, indistinguishable from a reading. The CI job `macOS energy backend (jcp-energy)` flaked for the same reason. Its test asserted the summed total advanced across a 200 ms CPU burn — satisfied by the GPU, not the CPU. It passed when something happened to draw on screen and failed when nothing did, which is why it failed overnight and "healed" by day. That flake has been blocking unrelated PRs. The fix enforces the contract where the rung is chosen. `detect()` selects IOReport only when `ioreport_status()` is `Live`: the CPU rail itself — `CPU Energy`, or `DIE_<n>_CPU Energy` on multi-die parts — must advance while this thread burns CPU, polled every ~25 ms for up to one second, probed once per process and cached. Liveness is decided on the raw integer before unit conversion, and never on the summed total, because the live GPU rail is exactly what corrupts the total. Otherwise the rung is declined and the time×power model estimates, as the contract requires. `IoReportStatus` (`Unavailable`, `CpuRailAbsent`, `CpuRailFrozen`, `Live`) is exposed so a caller or receipt can say why an Apple Silicon host is being estimated. The change is additive: `mod ioreport` is private and nothing outside the crate matches on `Meter`. Tests. The flaky test is replaced by three that assert against an independent oracle: it reads the CPU rail's ABSOLUTE cumulative value with no delta samples, matches names with `ends_with("CPU Energy")` rather than the implementation's die-prefix stripping, and burns for three times the probe's budget. There is no early return that passes without asserting — a host with no IOReport must report `Unavailable`, one with no CPU rail `CpuRailAbsent`. - `the_ioreport_rung_is_never_selected_on_a_frozen_cpu_rail` — the property that matters. It logs what it observed so a CI log shows which branch was asserted. - `the_liveness_probe_never_reports_a_frozen_rail_as_live` — the dangerous direction, asserted on its own. - `a_selected_ioreport_meter_meters_cpu_work` — when the rung IS selected, a CPU burn meters as non-zero energy. Observed on both machines: `status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false`, 10/10 passing — on the M3 Ultra via the `DIE_0_`/`DIE_1_` rails, which confirms the multi-die path. The CI job becomes deterministic: green while the meter correctly declines a frozen rail, red if it ever selects one. Falsified: a probe that always says Live fails the test (the oracle saw the rail hold); reverting `detect()` to select IOReport whenever it opens — main's behaviour — fails it with "IOReport selected although its CPU rail did not move", so main fails this test today. jcp-cli, jcp-conformance and jcp-intercept (every dependant of this crate) pass, as does the conformance suite's C9 energy-honesty check; clippy is clean on jcp-energy. jcp-gateway fails on main as well and does not depend on this crate — it meters through sandbox-meter's probe, which reads the same frozen rail. Not done here, deliberately: - A per-span check for a rail that freezes after `detect()` needs "CPU Stats" residency as a witness; the platform-level freeze observed here is caught at selection. - Reporting the live GPU rail as an explicitly partial reading, instead of declining, would need a coverage field on the intercept/1 wire — a format change, so a separate decision. - The same defect in jouleclaw-energy, eoc-meter, proof-cascade and sandbox-meter, and in the callers that seal a zero delta as a measurement. Co-Authored-By: Claude Opus 5.5 <[email protected]>The crate's contract, in its probe module doc: "A probe can degrade but it can never lie: if the hardware path fails mid-span, the reading comes back `Estimator`, not a zero dressed as a measurement." On current macOS the IOReport probe broke that on every Apple Silicon host. Ground truth, measured 2026-09-29 by dumping the Energy Model channels while burning CPU, not by re-running tests: - M5 Max, macOS 27.0.1: 364 channels. Of the three that `class_for` maps, `CPU Energy` (mJ) held at 480,083,160 and `ANE0` held across 3 s of burn; `GPU Energy` (nJ) advanced 682,543,073. Nothing else moved. - M3 Ultra, macOS 27.0.1: 645 channels, none moved, including `DIE_0_CPU Energy`, `DIE_1_CPU Energy`, `ANE0_0`, `ANE0_1`, `GPU Energy`. The defect. `IoReportProbe::new()` accepted the probe once a subscription opened and one snapshot was non-empty, and `Span::stop` sealed every class delta, zero included, as `measured_uj: Some(delta)` under `HwShunt`. main's own `joule-meter 1`, busy on all cores, sealed and verified: M5 Max: Cpu: 0 µJ [HwShunt] Npu: 0 µJ [HwShunt] Gpu: 226790 µJ [HwShunt] M3 Ultra: Cpu: 0 µJ [HwShunt] Npu: 0 µJ [HwShunt] Gpu: 0 µJ [HwShunt] On the M5 Max the display's GPU energy is passed off as the energy of CPU work; on the M3 Ultra, three measured zeros. The same figure flows on: `sample_uj()` sums the classes, and `run_attributed` apportioned a 0 J "measurement" with `attributed = true`. That is why dependants fail on main on these machines. This crate's `attributed_run_captures_a_busy_child` fails at attribute.rs:297 ("a busy window costs SoC energy"). joule-code-remediate's `verified_fix_end_to_end_ships_with_receipt` fails with `report.total_uj() > 0`: it apportions the frozen rail's zero. jcp-gateway's `granted_tool_is_allowed_metered_and_debited` (lib.rs:585, `measured_uj > 0`) and `budget_exhaustion_denies_further_calls_and_the_log_verifies` (lib.rs:638, "the budget eventually exhausts") assert, under `metering_is_hardware()`, that a span's summed total moved. That holds only while the GPU rail happens to tick: 40/40 green on the M5 Max this morning with the display compositing, red when it idles, and 3/3 and 2/3 red on the M3 Ultra. The fix applies the contract in the three places the decision is made. 1. Selection. `IoReportProbe::new()` declines unless `ioreport_status()` is `Live`. The CPU rail itself (`CPU Energy`, or `DIE_<n>_CPU Energy` on multi-die parts, matched after stripping the die prefix) must advance while the probing thread burns CPU. It is sampled every ~25 ms for up to `CPU_RAIL_LIVENESS_BUDGET` (1 s), probed once per process and cached in a OnceLock. The probe reads the same absolute samples `snapshot()` reads, on the probe's own subscription, so it tests the path a span will use. It decides per rail on the raw integer, before unit conversion, paired by position and name. It never decides on a summed total, which the live GPU rail advances by itself. Otherwise `new()` returns `ProbeError::Unavailable` naming the frozen `CPU Energy` rail, and `PowerMeter` falls back to its estimator: wall × TDP, `Estimator`, no `measured_uj`. A probe whose snapshot carries no CPU class is refused too, because GPU and ANE alone would be the same defect. 2. The span. In a span this process ran in, a CPU-class delta of 0 is a counter that held, not a reading. The whole span now returns the estimator record, exactly as when every probe vanished. That covers a rail that freezes after the per-process probe, and RAPL: a package counter that did not advance is no measurement either. A probe set with no CPU class (NVML alone) is not subject to it. I checked for tests that would break: none expects a short span to be `HwShunt`. jcp-gateway's hardware-only asserts need only a positive figure, which an estimate is. Whether the Linux runners' containers can read RAPL was not verified here. 3. Attribution. `run_attributed` sets `attributed` only when the window's CPU-class record is an `HwShunt` reading that advanced, on top of the existing rusage and busy-tick conditions. An estimate is carried as context and never apportioned as a reading. `IoReportStatus` (`Unavailable`, `CpuRailAbsent`, `CpuRailFrozen`, `Live`, with a Display), `ioreport_status()` and `CPU_RAIL_LIVENESS_BUDGET` are public and re-exported at the crate root, so a caller or CLI can say why an Apple Silicon host is estimated. `joule-meter` now does. Its "machine-wide counters; subtract idle" note is printed only for a measured span, because it is false for an estimate. Grade names are untouched: a live IOReport reading is still `HwShunt`. Tests. Every IOReport test asserts against an independent oracle. The implementation reads absolute samples on its own subscription and strips a die prefix. The oracle opens its OWN subscription, reads DELTA samples with `IOReportCreateSamplesDelta` (the implementation never calls it), matches `ends_with("CPU Energy")`, and burns for three times the probe's budget. No branch passes without asserting. - `the_ioreport_probe_is_never_selected_on_a_frozen_cpu_rail` checks the property and prints "ioreport liveness: status=.. oracle=.. selected=..". - `the_liveness_probe_never_reports_a_frozen_rail_as_live` checks the dangerous direction, and checks every other status claim against the oracle. - `a_selected_ioreport_probe_meters_cpu_work`: when selected, a CPU burn meters an `HwShunt` CPU record > 0. When declined, the status is not Live, the span is the estimator's, and `sample_uj()` is None. - `a_cpu_rail_is_the_rail_itself_on_one_die_or_many` and `liveness_is_decided_per_cpu_rail_on_the_raw_integer` cover the matcher and the pairing, using the dumps' names and values. - Scripted-probe span tests replay the dumps and are deterministic on every platform: `a_span_whose_cpu_counter_held_is_estimated_even_though_the_gpu_advanced` (M5 Max), `a_span_where_every_counter_held_is_estimated` (M3 Ultra), `a_held_cpu_counter_estimates_the_whole_composite_span` (RAPL held, NVML moved), `a_span_whose_cpu_counter_advanced_is_measured`, and `a_probe_set_without_a_cpu_class_is_not_subject_to_the_cpu_rule`. - `detect_never_fails_and_records_validate` used to assert `total_uj() > 0`, which the GPU rail satisfied. It now asserts the CPU record or the estimator record, and on Apple Silicon that the probe exists exactly when the status is Live. - `attributed_run_captures_a_busy_child` used to assert `soc_cpu_uj > 0` and fails on main. It now asserts per status: Live means attributed; otherwise not hardware, not attributed, and an estimator figure > 0. Two scripted tests pin the rule itself: `a_window_whose_cpu_counter_held_is_never_attributed`, and `a_window_whose_cpu_counter_advanced_is_attributed` for the positive path, which no macOS 27 host reaches for real. Their child now runs 2–3 s, bounded by the clock and an iteration cap. The machine tick counters behind `busy_cpu_secs` update in bursts up to ~990 ms apart on both chips. A ~0.3 s child saw `busy_cpu_secs = 0.0` on the M3 Ultra and would have declined for the wrong reason. Observed with the same test binary on both chips, 5 runs each, 31/31 every time: M5 Max: ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false M3 Ultra: ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false Falsified. Each mutation was restored with git checkout, leaving a clean tree: - F1, the probe always says Live: the frozen-rail test fails ("the oracle saw the CPU rail hold under 3s of CPU burn", left: Live, right: CpuRailFrozen), and so do the never-Live, metering and attribution tests. - F2, main's ungated `new()`: it fails with "IOReport probe selected although its CPU rail did not move under 3s of CPU burn — every span would report GPU-only energy, or zero, as a measurement of CPU work", and 3 others fail. - F3, the per-span rule removed: the three held-counter span tests fail. The M5 Max one reproduces main's seal exactly: Cpu Some(0), Npu Some(0), Gpu Some(682543), all HwShunt. - F4, main's attribution rule: `a_window_whose_cpu_counter_held_is_never_attributed` fails because the estimator's 27.8 J is attributed as a reading. Dependants, run on main and on this branch: sandbox-attest, -runtime, -exec-wasm, -exec-microvm, -conformance (and its suite: all classes, no skips), -proof, -proof-wasm, -mcp, -build, -cli, -jcp, jcp-gateway, jcp-action-receipt, joule-ui-interaction and joule-ui-serve pass on the branch. joule-code-remediate passed 15 of 16 branch runs. The failure was `verified_fix_end_to_end_ships_with_receipt` with the assertion it fails with on every main run, `report.total_uj() > 0`. On the branch that figure is estimate × share, and the share is 0 when no machine tick burst lands in any of its cargo windows. jcp-gateway passes because its hardware-only asserts are now skipped. Both relabel the estimate this crate now returns: see below. Clippy is clean on this crate (--tests), and so is x86_64-unknown-linux-gnu. Not done here, deliberately: - Consumers that relabel the estimate this crate now returns: jcp-gateway debits and seals `Energy::measured(measurement.total_uj())` (lib.rs:255-259) and labels its eval by `meter.is_hardware()` (lib.rs:228, 430). Its hardware-only asserts are now skipped on Apple Silicon. joule-code-remediate seals an estimate-apportioned figure as `ModelBased` (lib.rs:656), and its end-to-end test passes here only because `total_uj()` is now an estimate > 0. jcp-action-receipt (examples/end_to_end.rs:139), sandbox-proof (lib.rs:1321) and joule-ui label by `is_hardware()` rather than by the span. They are for the next change, on this branch. - A GPU or ANE zero beside a CPU counter that advanced, and an NVML-only set's zero GPU delta, are still sealed `HwShunt` 0 (probe/mod.rs:340). - Reporting the live GPU rail as an explicitly partial reading instead of declining the probe is a receipt and wire decision. - `class_for`'s substring match (ioreport.rs:101-111) sums every `DIE_n_CPU Energy`, which is right for an Ultra. It would also sum a sub-rail named "…CPU Energy", and it takes any channel containing "ANE". On these two dumps only the top-level rails match; other chips need a dump. - `sample_uj()` sums every class, so its before/after users cannot apply the held-CPU rule. `Attribution` has no provenance for `soc_cpu_uj`. Both are API changes. - `busy_cpu_secs` inherits the ~1 s tick bursts, so windows under ~1 s are declined or skewed. Co-Authored-By: Claude Opus 5.5 <[email protected]>The contract, stated in jcp-energy and sandbox-meter alike: measured when the platform exposes it, estimated when it doesn't, "and the provenance always says which". The parent commit makes sandbox-meter keep it. On macOS 27.0.1 no Apple Silicon host checked exposes a live CPU energy counter, so PowerMeter declines IOReport and every span is the estimator's wall × TDP record. On a live host, a span whose CPU counter held is estimated too. The meter's consumers then threw that provenance away. Ground truth: the consumers' own output at the parent commit. The same binaries ran on an M5 Max and an M3 Ultra, both on macOS 27.0.1; the M5 Max comes first. - jcp-gateway printed "metering=estimator", then "allowed, 1541728 µJ measured", and sealed energy_provenance "HwShunt" in the receipt. It debited Energy::measured(total_uj()) whatever the span was. On the M3 Ultra: "999293 µJ measured", sealed HwShunt. - joule-ui-serve sealed every interaction as Estimator with estimated_uj 0, for 24-56 ms of work. Its sampler summed raw counters under the Meter trait's default HwShunt. It returned None once no hardware probe was left, and run_interaction turns None into 0 J. - joule-fix printed "energy per verified fix: 1.446 J (attributed)" for an estimate apportioned by CPU share, and sealed it ModelBased in the JCR-1 meter envelope (M3 Ultra: 7.377 J, ModelBased). The share is 0 whenever the machine tick counters miss every window, so the figure can be 0 J. The parent commit gave that as the likely cause of verified_fix_end_to_end's 1-in-16 failure. - sandbox-proof said "ESTIMATED" here only because it asked is_hardware(). On a live host it would say "measured" for a span the meter estimated. The defect. Each consumer labelled a span by the meter instead of by the span's own records: is_hardware(), a trait default, or a hard-coded "ModelBased" or "measured". Or it turned a meter with no reading into 0 J. The fix. Every consumer derives the label from the Measurement it got: the weakest of its records, because a sum is only as good as its weakest part. - jcp-gateway. energy_of() builds the ledger entry with jcp-schema's own constructors: - Energy::estimator for the estimator's record; - Energy::measured for a hardware reading that has a CPU record; - Energy::measured_partial for a hardware reading without one (a GPU counter alone, NVML without RAPL); - Energy::modeled for ModelBased. The figure is unchanged, so a runaway is still debited. Only what the receipt says about it changes, through fields it already had: energy_provenance, and energy_coverage when Partial. The flow monitor's scan is sealed with its own span's provenance (flow_energy_provenance) instead of is_hardware()'s. The binary's summary line reads the grade back from the receipt it just sealed ("[Estimator]") instead of printing "measured". The docs and the crate description said "metered on real hardware" and "the MEASURED joules"; both are corrected. jcp-gateway now depends on sandbox-schema directly, for the CPU class; the crate was already in the lock graph. - joule-code-remediate. An EnergyTally grades the run by its cargo windows. The run is ModelBased only when every window was attributed. main needed only one, and never reset the flag between runs. An attributed run's receipts carry the attributed figure, anchored as before. Any other run's receipts carry the unapportioned total over the windows, as an estimate: - the JCR-1 meter envelope seals it with the Estimator grade; - the JouleContract receipt seals it as Unaccounted; - total_uj() and uj_per_verified_fix() return it. An apportioned estimate is no better than an estimate, and apportioning can put 0 J on the receipt for work that ran. joule-fix prints "estimated, not attributed" and "(estimated)" to match. - joule-ui-serve. SpanMeter replaces the summing sampler. Every sample closes the span the previous sample opened, adds its figure to a running counter, and opens the next. provenance() returns that span's own grade, which run_interaction asks for right after its second sample. An estimated span is sealed as Estimator with the estimator's figure, not 0 J, and a hardware span as HwShunt. The Meter trait is untouched. The binary's doc and its Cargo.toml said every receipt's HwShunt was true; corrected. - sandbox-proof. prove_local says "measured" only when every record of the span is a hardware reading (energy_word). The records were, and still are, sealed verbatim into the receipt, the attestation and the proof cell. - jcp-action-receipt. The end_to_end example sealed energy_provenance from is_hardware(); it now takes the span's. Checked and left alone: - sandbox-jcp debits a declared ceiling as Energy::estimator and never reads a PowerMeter. - joule-ui-interaction's library labels nothing as a hardware reading that its meter did not declare one. Its jcp_metered example wraps whatever figure it is charged in Energy::measured, but only in a JCP runtime decision that it neither seals nor prints. See below. After the fix, with the same binaries on both chips: - jcp-gateway prints "allowed, 2309714 µJ [Estimator]" and seals Estimator (M3 Ultra: 973555 µJ, Estimator). - joule-ui-serve seals 482410 and 1121739 µJ as Estimator for intensities 3 and 7 (M3 Ultra: 481236 and 1121485 µJ). - joule-fix prints "44.352 J estimated, not attributed" and seals Estimator and Unaccounted (M3 Ultra: 20.723 J). - sandbox-proof still says ESTIMATED. Tests. The same test binaries ran 5/5 green on each chip. On the M3 Ultra, joule-code-remediate's the_shipped_formal_template_parses_and_validates cannot find formal.example.json, whose path is fixed at compile time on the dev machine, so the five runs there skip that one test. That host also has no z3, so the two existing formal-proof tests return early there. - jcp-gateway a_metered_span_enters_the_ledger_as_what_it_is: every span shape maps to its constructor, and the debit is the span's own figure. - jcp-gateway a_calls_receipt_says_how_its_joules_were_obtained: end to end, the sealed energy_provenance is Estimator when the meter has no hardware source ("gateway call: hardware=false sealed=Estimator cost_uj=30621"). Otherwise it is a reading or an estimate. cost_uj > 0 either way. - jcp-gateway granted_tool_is_allowed_metered_and_debited and budget_exhaustion_denies_further_calls_and_the_log_verifies asserted only under metering_is_hardware(), so on these hosts they passed without asserting. They now assert on every host: an estimate is debited, and the budget exhausts. - The exfiltration test now also checks flow_energy_provenance. - joule-code-remediate an_unattributed_run_is_sealed_as_an_estimate_never_an_apportioned_zero and a_run_is_attributed_only_when_every_window_was pin the tally. - joule-code-remediate verified_fix_end_to_end_ships_with_receipt now also checks that joules_spent equals total_uj(), the JouleContract class, and the grade inside the sealed meter envelope. It also checks that nothing is attributed without a live CPU rail. - joule-ui-serve each_served_interaction_carries_its_own_spans_provenance replays a HwShunt span, then an Estimator span, through the real Host. - joule-ui-serve the_served_meter_never_seals_zero_or_an_estimate_as_a_reading runs the served meter on this host. - sandbox-proof a_local_proof_says_estimated_for_an_estimated_span. Falsified. Each mutation was restored with git checkout, leaving a clean tree. - energy_of returns Energy::measured(total), main's rule: both new gateway tests fail. The end-to-end one fails with left "HwShunt", right "Estimator". - eval_prov is sealed as HwShunt, which is what main sealed here: the exfiltration test fails with "an estimated scan sealed as a reading". Reverting eval_prov to is_hardware()'s answer is not caught on these hosts, because with no hardware source both say Estimator. - energy_of returns Energy::measured(0), as main did on the M3 Ultra: four tests fail, including the two formerly hardware-only ones ("a real span costs joules, measured or estimated", "the budget eventually exhausts"). - figure_uj returns the apportioned sum: three joule-code tests fail. The unit tests see 10000000 against 60000000; end to end, 1.373789 J sealed against 65.634219 J. - The meter envelope is always graded ModelBased: three tests fail. End to end: "an estimate sealed in the meter envelope as ModelBased". - main's any-window attribution: the mixed-window test fails. - main's total_uj, the apportioned figure: the end-to-end test fails (45.799836 against 0.776724). - SpanMeter without its provenance() override, so the trait's HwShunt applies: both joule-ui-serve tests fail. - served_meter returns the parent commit's HardwareMeter: the end-to-end test fails with "~24 ms of work sealed at 0 J: Energy { measured_uj: None, estimated_uj: 0, provenance: Estimator }". - energy_word always says "measured": the sandbox-proof test fails. Not separately tested: the per-run reset of the tally. It is observable only with attributed windows, which need a live CPU rail. Clippy --no-deps --all-targets is clean on every changed crate. It is also clean for x86_64-unknown-linux-gnu, except that sandbox-proof could not be checked for that target: zstd-sys's build script needs a Linux C cross compiler, which is not installed here. One pre-existing warning, in jcp-action-receipt's ar-verify.rs, is untouched. Not done here, deliberately: - The joule-ui Meter trait: its default provenance is HwShunt, and its provenance is per meter, not per sample. SpanMeter relies on run_interaction asking for provenance right after its second sample. run_interaction also turns a None sample into Estimator(0), so an Authorizer charges 0 for work that ran. Authorizer::charge carries no provenance. Both joule-ui-interaction examples still use the summing sampler. These are public API and SPEC §5/§6: David's call. - Wire keys named measured_uj now carry estimates: jcp-gateway's _meta and Relayed::Allowed, and joule-ui-serve's POST /action response. The sealed provenance beside each key is right. - A GPU counter alone (NVML without RAPL): the gateway seals it Partial but still debits only the GPU figure, so CPU work goes undebited there. Its two budget tests would fail on such a host, correctly. joule-ui has no coverage field. Not observed: no such host here. - JouleContract §4.2 says an Unaccounted figure seals its method reference (energy_anchor). The harness leaves it None, as it always has. - proof-cascade's joules: FormalOutcome, "(real z3, ModelBased)" in joule-fix, and formal_fix's "µJ measured". Another change owns that meter. - joule-code-remediate's Cargo.toml description still says hardware-measured. This commit does not touch that file. Co-Authored-By: Claude Opus 5.5 <[email protected]>The hardware tests for the CPU-rail liveness probe run in CI on the macOS runner, which is this M5 Max, and it is often heavily loaded: load averages of 24-104 have been seen. Round 3 left tests that only held on a quiet host, a cache that was pinned per thread rather than per process, and rail values nobody checked. This commit fixes those. The probe's tick - The tick now burns 25 ms of the probing thread's own CPU time, measured on CLOCK_THREAD_CPUTIME_ID. It gives up after 500 ms of wall time. - The old tick spun on the wall clock. On a loaded host the scheduler gave it only part of each 25 ms. With 36 extra spinners on this Mac, that old tick left intervals with only 8.6-11.8 ms of thread CPU. - The tick is still passed into watch(), so the scripted tests keep their scripted ticks. What the hardware tests assert now - The probe burns CPU. In every interval it counts, the probing thread must use at least half a tick (12.5 ms) of CPU time. Only the 500 ms wall cap could break that, and the assertion message says so. The probe records its thread CPU at each sample for this check. - The probe subscribes to the Energy Model group only. The median wall time of the probe's own sample calls must be under 50 ms. A group sample takes about 6-12 ms and an all-channel sample 130-800 ms. The probe records each call's duration, and the test prints the median. - The interval-count assertions are gone: ">= 10 intervals" and ">= 2 intervals plus the full budget". What stays is "a frozen or absent verdict took at least the 1 s budget". It is a lower bound, and load can only make the probe slower. No hardware test asserts a ratio of CPU time to wall time. - New test: the_probe_sampler_reads_each_cpu_rail_at_its_own_value. It checks the values the sampler reads, against an oracle on the oracle's own all-channel subscription. - The oracle reads every Energy Model CPU rail by name, once before and once after one probe sample. - If the oracle saw a rail hold, the probe's value must equal it. If the rail moved, the probe's value must lie between the two reads. - Before this, a sampler that returned constant 0, or gave each CPU rail the sum of all CPU rails, passed every test. The oracle - It keeps reading past its 3 s window, up to 4 windows, until it has at least 2 reads after its baseline. - With fewer reads it fails the test, with the read count and the reason. It never returns CpuRailMoved(false) on 0 or 1 reads. - The "ioreport liveness:" line now ends with oracle_reads=N. The test that prints it keeps its name. - Under load the oracle needed the extension: one run got 2 reads in 3.76 s. The pure rule: each case has its own test - A rail that goes down between two samples (a reset or wrap) has not advanced. - A rail missing from the previous sample only sets its baseline. - No verdict maps to Unavailable. - A probe that cannot subscribe to the Energy Model group, or whose subscription never returns a sample, maps to Unavailable through status_of_run(). It is never Live and never CpuRailAbsent. That last rule changes behaviour. A host where the group cannot be subscribed but the all-channel subscription opens used to report CpuRailAbsent. It now reports Unavailable, because the probe saw nothing. IoReportStatus's documentation says this. The enum and its variants are unchanged. The once-per-process cache - A process-wide AtomicUsize, HOST_PROBE_RUNS (built only for tests), counts the runs of the real host probe. The thread-local counter is removed. - the_liveness_status_is_probed_once_and_cached now starts 6 threads through a barrier. Each calls ioreport_status() twice, builds two counters and calls detect_meter. The test asserts the host probe ran exactly once in the process. - This corrects073fb7d8, which said the cache was pinned "once per process". Its thread-local counter only pinned "once on this thread": a per-thread cache passed all 50 tests. Evidence - M5 Max (macOS 27.0.1), clean test binary, 10 runs under 36 background spinners (2 x hw.ncpu). Load average went from 34.8 to 142.8 during the runs. The spinners were killed afterwards, and none was left. - Result: 10/10, "test result: ok. 55 passed". - Every run: "ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false oracle_reads=3..7". - Group probe: 5-10 intervals, median sample cost 7.16-10.24 ms. - Least thread CPU in any interval: 31.72-32.39 ms. - Oracle: 2-8 reads after the baseline. - M3 Ultra (27.0.1), the same binary: 55 passed. - Median sample cost 6.75 ms over 33 calls. - Rails DIE_0_CPU Energy and DIE_1_CPU Energy. - Sampler values equal to the oracle's: 28237793 and 30845042. - Oracle: 16 reads. Mutations. Each was a temporary edit, restored with git checkout, and git status was clean afterwards. - A decrease counts as an advance: 2 tests fail, "a fall counted as an advance". - A rail new in a sample counts as an advance: 1 fails, "a new rail counted as an advance". - No group subscription maps to Live: 1 fails. - Per-thread cache: 1 fails, "the host liveness probe ran 13 times in this process". With no cache at all, 41 times. - The sampler returns constant 0: 1 fails, "held at 1863990848 ... but the probe's sampler read 0". - Each CPU rail gets the sum of all CPU rails: - On the M3 Ultra it fails, "("DIE_0_CPU Energy", 0) held at 28237793 ... but the probe's sampler read 59082835". - On the M5 Max it is an equivalent mutant, because that group has one CPU rail. Summing every channel named *CPU* into it does fail there: "read 5567863491". - The tick sleeps: fails, "interval 0 of 29 burned only 6.086583ms". A no-op tick fails the same way: "interval 0 of 114 ... 8.550958ms". - The probe on the all-channel subscription: fails, "median sample call took 538.447416ms". - The oracle concludes after 1 read: 2 tests fail with the read-count message. - The mutations whose old assertions were removed are still caught: - watch() stopping after 1 interval fails 4 tests. - A 10x-fast probe clock fails, "decided CpuRailFrozen after 211.462541ms, before its 1s budget". - PROBE_TICK = 250 ms fails the scripted test (a). - The round-3 wall-clock tick passes on a quiet host. Under the same load it fails 3/3 on the new CPU pin. Other checks - No features: 24 passed. - Linux x86_64 and aarch64, with and without ioreport: cargo check passes. - Clippy: no new warnings. - Consumers: - jouledesk-github: 20/20. - jouledesk: 18 + 4. - loader-gguf with rayon and fabric-meter: 302 passed. Runs with only fabric-meter hit a tier_store temp-file race (NotFound) twice in 12 runs, in a file this branch does not touch; the 10 other runs passed. - harness-econ: 14. - mesh-meter with hw-meter-apple: 10. mesh-core: 3. - l3-jouleclaw: 2 + 4. Not done here - The counter's own read() still samples every channel. It is declined on both Macs today. - No CI job runs these tests: .github/workflows is off-limits to this branch. Co-Authored-By: Claude Opus 5.5 <[email protected]>Round 3 left three mutations alive and one assertion whose margin shrank with load. This Mac is also the macOS CI runner, often at load averages of 24-104, so every pin on how the probe paces itself had to stop depending on wall time. This commit closes all four. What changes in the probe: - A tick now burns thread CPU time. It spins until the thread's own clock, clock_gettime(CLOCK_THREAD_CPUTIME_ID), has advanced 25 ms, with a 500 ms wall-clock cap. Before, a tick spun 25 ms of wall time, and on a loaded host that is a few milliseconds of CPU. Under 36 extra spinners, the old tick got a median of 3.2 ms of CPU in 48 ms. - The probe records the wall time of each of its own FFI sample calls, and the CPU and wall time of each tick. Tests read how it sampled and burned from that record, not by timing it from outside. - With no Energy Model subscription, the status is now Unavailable, through the same pure status_of as a probe that saw no interval. Before, a failed group open beside an openable counter said CpuRailAbsent, which claims a group was read and had no CPU rail. probe_with() is generic over the opener, so that path now runs in a test. Tests: - Group-only subscription: the MEDIAN wall time of the probe's own sample calls is under 50 ms. Group samples measured 5.7-12.8 ms; all-channel ones 142-714 ms. This replaces the >= 10-interval floor. No hardware test asserts a wall-clock ratio. - Burning: thread CPU per interval is at least 12.5 ms by the tests' own clock (clock_gettime_nsec_np); only the 500 ms wall cap could cut it short, and the message says so. From the probe's record, every tick burned 25 ms of thread CPU or hit the cap, and a tick came before every re-sample. - New: the_probe_sampler_reads_each_cpu_rail_as_the_oracle_does. The oracle reads each Energy Model CPU and GPU rail absolutely, on its own all-channel subscription, before and after each of two probe samples. A rail it saw hold must read EQUAL in the probe's sample; a rail that moved must read between the two. This catches a sampler that returns constant 0 (round 3's X3), sums rails, or reads the wrong channel. - Pure-rule pins, one test each: a decrease is not an advance (X5); a rail missing from the previous sample only sets its baseline; no subscription, or no sample, is Unavailable (X4); no verdict is Unavailable. - Oracle evidence: fewer than two reads after the first is now a failure (Oracle::TooFewReads), never CpuRailMoved(false). The oracle extends its 3 s window up to 5x to get the two reads. - The cache pin is process-wide. Eight threads, released together by a barrier, ask for the status and build the meter. The static PROBE_RUNS counter must read exactly 1. - The IOReport hardware tests now share one lock. Side by side, one test's all-channel reads slowed another's group samples: max sample calls reached 132-279 ms, and 12 probes at once pushed the median to 112 ms. Serialised, the median is 6-13 ms. The suite takes ~12-16 s instead of ~5 s. - `the_ioreport_rung_is_never_selected_on_a_frozen_cpu_rail` still prints "ioreport liveness: status=". It now shows how long the oracle actually read. Evidence, on this exact tree: - M5 Max, 10 runs of the lib tests with 36 extra spinners (2 x hw.ncpu), at 1-minute load averages of 61-180: 31/31 in all 10. Median sample cost was 7.0-12.8 ms, thread CPU per interval 32.3-33.6 ms, CPU per tick 25.00 ms, and the oracle took 5-17 reads. Every run printed status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false. - M3 Ultra, same binary: 31/31. DIE_0 and DIE_1 read equal to the oracle (28237793 and 30845042), and the median sample cost was 5.9 ms. - Each mutation below was restored with git checkout, and each one fails a test: X3 constant-0 sampler, X4 no subscription -> Live, X5 decrease counts, a new rail counts, per-thread cache, sum of CPU-named rails, GPU value given as CPU, tick sleeps, tick is a no-op (both the burn call and the closure), tick burns wall time, probe on the all-channel subscription, no OnceLock, compare with the first sample, TimePower{0} when declined, None -> Live, and a coarse threshold. Summing the CPU Energy rails is the identity on the M5 Max's single rail, so it cannot fail there. On the M3 Ultra it fails: "held at 28237793 ... but the probe's sampler read 59082835". - jcp-cli, jcp-conformance (lib and pack) and jcp-intercept pass. So does the conformance suite, including "C9 energy provenance honesty", on both machines. clippy --no-deps is clean for aarch64-apple-darwin, x86_64-apple-darwin and x86_64-unknown-linux-gnu, with and without --tests. Not done here, deliberately: - C9 still meters `|| ()`, so a declined meter giving 0 passes it. That is a conformance-suite change. - The standards-build.yml comment is stale, and its job lacks --nocapture. PR #102 owns that workflow. - read_uj still samples every channel. counts_once admits CPU sub-rails that do not contain "cpu" (e.g. PACC_*) beside CPU Energy, a double count once a rail is live. The receipt carries no IoReportStatus (a wire decision). CPU_RAIL_LIVENESS_BUDGET is public on macOS only. Co-Authored-By: Claude Opus 5.5 <[email protected]>The liveness probe's hardware tests run in CI on the macOS runner, which is the M5 Max this was written on, at load averages of 24-104. The parent commit's tick spun for 25 ms of WALL time, so on a loaded machine each tick burned only the CPU the scheduler gave the thread. Its tests pinned the burn and the group-only subscription with load-dependent bounds: burned CPU at least a quarter of the ticks, and a Frozen verdict on at least 10 intervals. With the tick below, the probe completes 11-13 intervals in its 1 s budget under 2 x ncpu busy-loop spinners (29 on an idle machine), so that floor would have sat one to three intervals from failing. The tick (probe/liveness.rs): - spin(cpu, wall) burns until the thread's CPU clock has advanced PROBE_TICK (25 ms), with a 500 ms wall cap per tick. The real probe passes CLOCK_THREAD_CPUTIME_ID (thread_cpu in ioreport.rs) and a monotonic clock. A loaded machine makes a tick longer, never lighter. - Both clocks are injected, so the rule is tested with scripted ones: a full tick on a third of a core, and a thread that gets no CPU ending at the cap. The loop still takes an injected tick, and the scripted-loop tests are unchanged. What the probe records as it runs (ProbeTrace, returned beside the RailWatch): - the wall time of every FFI sample call; - per interval, the thread CPU its ticks burned, measured around the ticks, apart from the samples. The hardware test the_liveness_probe_never_reports_a_frozen_rail_as_live now asserts: - every interval burned at least PROBE_TICK/2 (only the wall cap can cut a tick short, and the message says so); - the median sample call took under 50 ms (a group sample is ~6-12 ms, one over every channel 0.13-0.8 s), and it prints that median. The ">= 10 intervals" floor, the "burned >= a quarter of the ticks" bound and the ">= 2 intervals for Frozen" check are gone. The cap check stays, with generous slack (5 budgets + 5.5 s). The sampler's values, against the oracle. The new test the_probe_sampler_reads_each_cpu_rails_own_value brackets each of three probe samples between two oracle reads on the oracle's own all-channel subscription. It matches each rail by name and ordinal in the Energy Model group: - where the oracle saw the rail hold, the probe's value must equal it; - where the rail moved, the value must lie between the two reads. This catches a sampler that returns 0, a sum over rails, or another rail's value. the_probe_samples_the_energy_model_group_alone also checks every sampled name with is_cpu_rail, so the FFI filter is pinned on any host (round-3 nit). The oracle reads for its window, and on past it until it has two reads, never past four windows. Fewer than two reads is a test failure with a message, never CpuRailMoved(false). Its load thread runs until the oracle stops reading. Pure pins, each its own test: - a decrease between consecutive samples is not an advance; - a rail new in a sample only sets its baseline; - probe_host is the open-watch-map path the real probe now goes through. A subscription that cannot open, or never yields a sample, goes through status_of to Unavailable, and is never watched; - status_of(None) is Unavailable. The once-per-process cache is pinned process-wide. Six threads, released by one barrier, ask for the status and build the backend at once, and the static PROBE_RUNS counter must read 1. The test that prints "ioreport liveness: status=" keeps its name, and no test was renamed. Evidence, from one lib test binary of this tree (55 tests): - M5 Max (macOS 27.0.1), 10 runs under 36 busy-loop spinners (2 x hw.ncpu), load averages 124-149 (other sessions included). 10/10 passed. Every run printed: 'ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false' - median sample cost: 6.93-9.01 ms; - intervals per probe: 11-13; - least burn in one interval: 25.0009-25.0015 ms; - oracle reads: 7-12. Afterwards all 36 spinner PIDs were confirmed gone. - M3 Ultra (macOS 27.0.1), 1 run: 55 passed; median 8.28 ms over 30 calls; 29 intervals; least burn 25.0003 ms; 17 oracle reads; both dies equal to the held oracle values. Mutations, each restored with git checkout (git status clean after each). Every one fails a test on the M5 Max except the one marked: - a decrease counts as an advance: 2 failed; - a rail new in a sample counts as an advance: 2 failed; - no subscription maps to Live: 1 failed; - per-thread cache: 1 failed ('left: 16 right: 1'); - sampler returns 0: 1 failed ('held at 1863990848 ...; the probe read 0'); - sampler reports the GPU rail as CPU Energy: 5 failed; - FFI filter also keeps the GPU rail: 6 failed; - spin sleeps: 4 failed ('a 25ms tick used 8.875µs'); - the real tick sleeps: 2 failed ('interval 0 of 26: the probe burned 8.541µs'); - spin is a no-op: 4 failed; - the real tick is a no-op: 2 failed; - the tick counts wall time: 3 failed; - all-channel subscription: 2 failed ('median sample call took 481.672458ms'); - None maps to CpuRailAbsent: 4 failed; - oracle cut to 1 ms: 2 failed ('the oracle got 1 read(s)'); - PROBE_TICK = 100 ms: 1 failed; - sampler gives each CPU rail the sum of all CPU rails: EQUIVALENT on the M5 Max, which has one CPU rail (55 passed). On the M3 Ultra it fails: 'DIE_0_CPU Energy #0 held at 28237793 ...; the probe read 59082835'. Co-Authored-By: Claude Opus 5.5 <[email protected]>Round 4 left one medium finding and four mutations that no test caught. This commit closes all five. The hardware tests run in CI on this M5 Max, which is often heavily loaded, so every new pin reads the probe's own record or a count. None of them is a wall-clock ratio. The once-per-process pin counts the real probe - PROBE_RUNS (test builds only) is now incremented inside probe(), the function that opens the Energy Model subscription and samples it. Round 4 counted in probe_status, the wrapper that the cache called. So a counter constructor that called probe() directly, uncached, passed all 55 tests (O6). - probe_status is gone; host_status caches probe() itself. - Tests that run the probe on purpose, uncached, call the same probe() inside deliberately(). That is a test-only scope which marks one call, on one thread, as not the host's run. It exists only in test builds, so no production path can use it and still build. - the_liveness_status_is_probed_once_and_cached still releases 6 threads through a barrier. Each thread calls ioreport_status() twice, builds two counters and calls detect_meter. The real probe must have run exactly once in the process. A constructor that probes uncached now fails it with "the real liveness probe ran 26 times". The tick is pinned from the probe's own record - probe_cpu_rail records each tick: the probing thread's CPU time, on its own CLOCK_THREAD_CPUTIME_ID, and the wall time. Every tick must show a whole PROBE_TICK (25 ms) of thread CPU or have hit the 500 ms wall cap, and there must be one tick before every re-sample. - Round 4 pinned only the CPU used per interval, from the end of one sample to the end of the next. That includes the sample call's own 6-8 ms, so a tick that burned 25% or 30% of PROBE_TICK passed under load (O1, O1b). Both now fail at any load, for example: "tick 0 of 66 burned 6.251291ms ... a tick burns 25ms of thread CPU time". - The record is checked against the tests' own thread clock, which is Mach thread_info, not clock_gettime. The thread must have used at least as much CPU as the probe says its ticks and its counted intervals used. A probe whose "thread CPU" clock was really the wall clock passes this only when the thread was almost never off the CPU. Under load it fails: "its ticks used 804.630251ms ... the thread used only 566.975ms". The no-subscription path goes through the pure mapping, and a test runs it - probe() is now probe_with(open, probe_on), given the real opener and the real probe. probe_with maps "nothing opened" through status_of_run to Unavailable, the same as a probe that saw no interval. - A test-only switch, NO_ENERGY_MODEL_GROUP, makes open_energy_model return None on one thread. a_probe_that_cannot_subscribe_to_the_group_is_unavailable runs the real probe() that way, on a host whose group opens, and asserts Unavailable with nothing probed. It also calls probe_with with an opener that opens nothing, and with a subscription that never returns a sample. - Round 4's O2 (probe() maps a failed subscription straight to CpuRailAbsent) passed on both Macs. It now fails with "the real probe with no Energy Model subscription ... left: CpuRailAbsent". The same bypass returning Live fails too, placed in probe() or in probe_with. The sampler is pinned to read afresh - New test: the_probe_samples_afresh_across_its_window. While the real probe runs, an oracle thread reads the Energy Model group every 10 ms and records when each read ran. It uses its own subscription, opened with the raw FFI, not open_energy_model. The probe records each sample call's start, end and values (test builds only). - For every probe sample with an oracle read wholly before it and one wholly after it: a GPU Energy rail that held must read the oracle's value, and one that moved must read between the two oracle reads. - If the oracle saw a GPU Energy rail change between two reads strictly inside the probe's window, the probe's own samples must show it changing too. - Fewer than two oracle reads inside the window is no evidence, and fails. - the_probe_sampler_reads_each_cpu_rail_at_its_own_value now holds GPU Energy to the same bracketing rule as the CPU rails, over two rounds on one subscription. - Round 4's O3 (the sampler returns its first sample's values forever) passed on both Macs. It now fails. So do a sampler that is stale per probe run and one that is stale per subscription. Example: "probe sample 1: ("GPU Energy", 0) moved 123056146417043 -> 123056172426143 across the oracle's reads either side of it, but the probe's sample read 123056142245337, outside them: not a fresh sample". - What this test can show depends on the GPU: - The CPU rails are frozen on both Macs, so they never can. - On this M5 Max, GPU Energy changes every ~28 ms. It moved inside the window in every run here. - An M3 Ultra's GPU moves only sporadically: in 113 of 11,716 100 ms samples over 25 minutes one day, and in about a fifth of 30 ms intervals today. Today the M3 Ultra caught all three stale samplers. - In a window where the GPU does not move, the test prints "no GPU rail moved inside the window: this run cannot tell a fresh sampler from a stale one", and a stale sampler would pass that run. Docs: the budget's doc no longer says "each still under a full tick of burn", and the all-channel sample cost is now quoted as 0.1-1 s. Evidence, on this exact tree - M5 Max, macOS 27.0.1, 10 runs under 36 background spinners (2 x hw.ncpu), 1-minute load average 23-57: 10/10 "test result: ok. 57 passed". All spinners were killed afterwards and none was left. Every run showed: - "ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false", with oracle_reads=8-13. - "the real probe ran 1 time(s)". - Probe: 7-12 intervals, median sample cost 6.98-7.79 ms, least tick 25.0005 ms of thread CPU, least interval 31.34 ms. - Freshness: the oracle saw GPU Energy change 43-52 times inside the window, and the probe's own samples changed in every interval (6-13 changes). - M5 Max, 5 more runs under 72 spinners (4 x hw.ncpu), load up to 75: 5/5. - M3 Ultra (10.10.10.2), the same binary: 57 passed. DIE_0 and DIE_1 read equal to the oracle (35656018 and 38872869). GPU Energy moved 9 times inside the probe's window, and the probe's samples 9 times. - 31 mutations. Each was applied, built, run, and restored with git checkout, with git status clean after each one. - These fail at least one test: baseline never moves; same-named rails counted under one key; a decrease, or a first appearance, counted as an advance; no subscription mapped to Live (in status_of_run, in probe_with, in probe); no subscription mapped to CpuRailAbsent (in probe_with, in probe); per-thread cache; no cache; constructor probing uncached; constant-0 sampler; three stale samplers; tick sleeps; tick is a no-op (the burn call, or the whole tick); tick burns 30% or 25%; tick burns on a helper thread; all-channel subscription; liveness decided on the sum of every Energy Model rail, or on the GPU rail; advances summed over rails; a declined counter reading 0; the fallback metering 0. - Two pass on a quiet M5 Max, and each is caught where it differs. The first gives each CPU rail the sum of all CPU rails. It is equivalent on the M5 Max's single rail, and on the M3 Ultra it fails: "DIE_0_CPU Energy held at 35656018 ... but the probe's sampler read 74528887". The second makes the wall clock the probe's thread clock. It agrees with the thread clock on a quiet host, and fails 3/3 under 36 spinners. - 24 no-feature tests pass. cargo check --tests --examples passes for aarch64-apple-darwin, x86_64-apple-darwin, aarch64-unknown-linux-gnu and x86_64-unknown-linux-gnu, with and without ioreport. clippy --no-deps shows only the warnings that were already there. Not done here - A path that re-implements probe(), by calling probe_with or probe_cpu_rail directly, is not counted. - No CI job runs these ioreport tests; .github/workflows is outside this branch. Co-Authored-By: Claude Opus 5.5 <[email protected]>The tier_store tests sometimes failed with "tier_store.rs:738:50: called `Result::unwrap()` on an `Err` value: Io(Os { code: 2, kind: NotFound, ... })". write_synth_gguf named its file jouleclaw-tier-store-<pid>-<nanos>.gguf. On macOS SystemTime ticks in microseconds, and the harness starts these tests together on parallel threads, so two tests could pick the same name. They then shared one file, and the first to finish deleted it before the other had read it. Each such helper now takes a number from its own process-wide AtomicU64 instead of the time, giving <prefix>-<pid>-<seq>. The crate's mellum, falcon_h1 and hybrid_lm tests already name their files this way. Four more helpers used the same timestamp naming with no per-test tag, and several tests call each of them at once, so they get the same fix: - lib.rs prefetch_policy_tests::write_synth_gguf (8 tests) - gpu_buffer.rs aligned_temp_mmap (5 tests; it truncates and maps the file) - mmap_advise.rs tempfile_compat (2 tests) - moe.rs build_synthetic_store_with_experts (2 tests) None of the four failed in the runs below, but they had the same race. Only #[cfg(test)] code changes. Evidence, on this M5 Max: - Before, on the parent tree (3b260e0a): the lib test binary built with --features fabric-meter, run 30 times, passed 27 and failed 3. All three failures were at tier_store.rs:738:50 with NotFound, in pin_hot_changes_tier_to_hot, demote_to_cold_returns_to_cold and out_of_bounds_errors. - After, the same build and the same 30 runs: 30/30, 299 passed each time. - `cargo test -p jouleclaw-loader-gguf --lib --features fabric-meter`, 12 consecutive runs: 12/12, "test result: ok. 299 passed; 0 failed; 2 ignored". - With --features "rayon fabric-meter": 3/3, 302 passed. - clippy --no-deps --tests --features fabric-meter: no warning on or next to any changed line. It exits 101 on three error-level lints that were already there, in files this commit does not touch: ssm.rs:406 (approx_constant) and gemma4.rs:562 and :582 ("this operation will always return zero"). Co-Authored-By: Claude Opus 5.5 <[email protected]>Round 5 left two dangerous survivors. Tests only ever saw CpuRailFrozen, the one status either Mac produces, so these two passed 57/57 on both Macs: - OWN1: a counter that declined only a frozen rail. - OWN4: a host cache that took "no subscription" for a live rail. It also left four fail-safe survivors: - a constructor that re-implemented the probe, which went uncounted; - a zero or stale transform between the probe's record and the rule; - a tick that always ran to its wall cap; - a latent flake on a host whose rail is live. In the sibling crates, a wall-time median pin also failed on correct code under concurrent all-channel IOReport sampling. This commit closes all of these. The production chain - probe_host_status() is the one production status. It bumps REAL_RUNS, then returns status_of(probe_with(open_probe_sampler, probe_tick, <monotonic clock>, CPU_RAIL_LIVENESS_BUDGET)). It has no branch of its own. - probe_with opens a sampler and watches it. If nothing opened, or no interval was observed, it returns None, which status_of maps to Unavailable. - crate::ioreport_status() now holds the process-wide OnceLock. It is the only production caller of probe_host_status. - AppleIoReportCounter::new() is new_given(ioreport_status()). new_given declines, before opening anything, unless admits(status) says yes. admits is a pure, exhaustive match that is true for Live alone. - GROUP_OPENS counts every call of the real group opener. Only the probe opens the group. So a constructor that re-implements the probe around the cache shows up there, even though it bypasses REAL_RUNS. Both counters are read through #[doc(hidden)] ioreport_probe_runs() and ioreport_group_opens(). Every status is tested - admits: Live admits; Unavailable, CpuRailAbsent and CpuRailFrozen do not. - new_given with Unavailable, CpuRailAbsent or CpuRailFrozen: - opens no subscription of any kind and samples nothing; - is unavailable and names the status; - read() returns NoCounter. - new_given(Live) opens the all-channel subscription once and probes nothing. - OWN1 now fails: "a counter given Unavailable opened a subscription or sampled: Record { group_opens: 0, all_channel_opens: 1, ... }". - The once-per-process pin now lives in its own integration test, tests/ioreport_probe_once_per_process.rs, so it runs in a process of its own. A barrier releases 9 threads: - a third ask for the status first; - a third build a counter first; - a third call detect_meter first. REAL_RUNS and GROUP_OPENS must both end at 1. Without the ioreport backend the test asserts that nothing probed. - In the lib test process the uncached status now runs on purpose, so the unit test there only checks that every thread sees the one cached status. Fault injection inside the production code - A test-only, thread-local seam sits in four places: - inside the real group opener; - inside the all-channel opener; - inside the FFI leaf of the real sampler, sample_raw, now split from the pure energy_model_channels filter; - at the top of RailWatch::observe. - A test can script whether the group opens and what each sample holds. Everything between the leaf and the rule is production code, including the real 25 ms thread-CPU tick and the real clock. - The real, uncached probe_host_status runs under these scripts: - The group does not open: Unavailable, one open call, zero sample calls. OWN4 now fails: "the production status with no Energy Model subscription". - The M5 Max's own single refresh: CpuRailFrozen. - A rail advancing twice: Live, after exactly 3 sample calls. - A constant rail with one failed sample: CpuRailFrozen. The failed sample is kept from the rule, and the verdict comes only after the whole budget. - Two dies stepping in different intervals: CpuRailFrozen. The script also has same-named sub-rails and an out-of-group channel named "CPU Energy" that moves in every sample. Each die reaches the rule at its own value. - No CPU rail: CpuRailAbsent. - Every scripted run also checks two things: - The rule was handed exactly the Energy Model channels of each sample that came back. - Each tick between two sample calls burned at least 25 ms and under 50 ms of this thread's CPU on the tests' clock, unless it hit the 500 ms wall cap. What the rule is handed - The record is now taken at the rule's input (observe), not beside the sampler. A zero or stale transform anywhere between the FFI leaf and the rule now shows up in it. - The freshness test runs the real uncached composition while an oracle reads the group on its own subscription. It checks: - the rule's input equals what the sampler read; - its CPU and GPU rails are the oracle's; - each bracketed sample equals the oracle where the rail held, and lies between the oracle's reads where it moved; - a rail the oracle saw change inside the window also changes in what the rule was handed. - On a host whose rail is live, the probe stops after two intervals (~60 ms), and one such window often holds too few oracle reads. So the test runs the composition up to 8 times, until one run has evidence and a moving rail. - No evidence in 8 runs fails. - Evidence but no moving rail passes, and the test prints that such runs cannot tell a fresh sampler from a stale one. Cost and clocks, with no wall-clock bound - The group-only pin is structural. Every channel of every sample the real probe took is in the Energy Model group. Each sample holds exactly as many channels as the oracle's own group subscription: 364 on the M5 Max, 645 on the M3 Ultra. - The cost pin is the median THREAD CPU per sample call, under 50 ms on the tests' clock. The wall-time median is gone. - No hardware test asserts a wall-clock bound or ratio. The one wall-clock check left is a lower bound in a scripted test: a Frozen verdict comes only after the whole budget. - The tests read their own CLOCK_THREAD_CPUTIME_ID, the literal 16 from the SDK's <_time.h>. Against it, the production clock: - does not advance over a 200 ms sleep (10.5 µs); - does not advance while a helper thread burns 200 ms (57 µs); - advances by what the tests' clock saw over this thread's own burn (100.001 ms for 100.0008 ms). - Each production tick burns at least 25 ms and under 50 ms of this thread's CPU (unless it hits its 500 ms cap), and no more CPU than wall time. - The probe's own record of its ticks and sample costs is gone: ProbeRun, Tick, DELIBERATE_PROBE, NO_ENERGY_MODEL_GROUP and PROBE_RUNS. The tests now measure the ticks themselves, between sample calls. counts_once - Ported from eoc-meter7aa42418. Beside the CPU aggregates, counts_once now also drops: - per-cluster accumulators (PACC*, EACC*, MACC*); - CPU power managers (*CPM*), with their _SRAM rails. It matches on the die-stripped name. The test pins the real channel names of both chips. Evidence (this commit's tree; M5 Max, macOS 27.0.1, hw.ncpu=18) - Tests with --features ioreport: - lib: "test result: ok. 66 passed; 0 failed" (57 before); - ioreport_probe_once_per_process: 1 passed, "real probe runs=1, Energy Model group opens=1"; - vendor_energy_conformance: 1 passed. - Tests without the feature: lib 24, integration 1, conformance 1. - Every probe status line said "ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false". - Load runs, on the lib test binary built from this tree (sha256 df3f59b0…): - 5 runs while a helper process sampled IOReport over all ~11,900 channels in a tight loop (~185 ms a sample): 5/5, each "66 passed". Median sample-call thread CPU 6.95-8.65 ms, median tick 25.071-25.079 ms. - 5 runs under 18 busy-loop spinners (1 x ncpu, load1 14.2 -> 27.3): 5/5. Median sample-call thread CPU 6.59-6.82 ms. - Every helper was killed; kill -0 and pgrep found 0 left. - A light group-only watcher of mine was also running during these runs. - M3 Ultra (10.10.10.2, 27.0.1), the same binaries, run with --nocapture and then deleted: - lib "test result: ok. 66 passed; 0 failed", and the once test "real probe runs=1, Energy Model group opens=1". - 645 channels per sample, equal to the group; median 7.10 ms; DIE_0 and DIE_1 keyed apart. - GPU Energy changed 6 times inside the window and 7 times in what the rule was handed: fresh. - Simulated live host (a temporary edit makes is_live true after two intervals; restored with git checkout), freshness and group-only tests only: - 10/10 unloaded, 10/10 under the all-channel helper. - Under the helper, the median thread CPU over the 3 sample calls was 6.1-8.3 ms; round 5's wall-time median was 100-202 ms there. - Every ~57 ms window held 3-4 oracle reads. - Where no rail moved in any of the 8 windows, the test said it cannot tell fresh from stale: 4 of 10 unloaded, 6 of 10 under the helper. In this simulation the CPU rail is really frozen. On a genuinely live host the rail itself moves in nearly every interval. - Mutations: 22, each an exact-string edit of this tree, run with cargo test --no-fail-fast (lib and integration), then restored with git checkout (clean after each). All 22 failed: - new_given declines only CpuRailFrozen: "a counter given Unavailable opened a subscription or sampled". - no subscription -> Live inside probe_host_status: "the production status with no Energy Model subscription". - a single advance -> Live inside the production closure: "one batch refresh counted as a live rail" and "two dies' separate steps ... made a live rail". - the rule compares with the first sample: 11 tests fail. - a decrease counts as an advance: "a fall counted as an advance". - a no-op tick: "production tick 0 burned 416ns ...". - a tick that always runs to its wall cap: "production tick 0 burned 455.483167ms ... a tick stops once it has burned 25ms". - a per-thread cache: "the real probe ran 10 time(s) and asked for the Energy Model group 10 time(s)". - a constructor that probes uncached: "ran 19 time(s) and asked ... 19". - a constructor that re-implements the probe through probe_with: "ran 1 time(s) and asked for the Energy Model group 19 time(s)". - a zero or stale transform in the production sampler closure: "the rule was not handed exactly the Energy Model channels of each sample". - a zero transform inside watch() before observe: 7 tests fail. - zeroing inside the rule after the record: 6 tests fail. - the sampler reads constant 0: "("CPU Energy", 0) held at 2643843378 across the oracle's reads either side of it, but the rule was handed 0". - the thread clock is CLOCK_MONOTONIC: "this thread slept 200ms, and the production thread clock advanced 210.015ms". - admits accepts CpuRailAbsent: "CpuRailAbsent admitted the counter". - counts_once re-admits PACC_*: "PACC_0, a summand of CPU Energy on an M5 Max, is summed beside it". - extras: - CLOCK_PROCESS_CPUTIME_ID: "advanced 1.767591s". - the probe on the all-channel subscription: 9 lib tests and the integration test fail. - each CPU rail handed to the rule as the sum of all CPU rails: fails on this one-rail M5 Max through the two-die script. - a tick of 30%: "burned 7.50375ms". - Consumers, all passing: - jouledesk-github: 10/10, 12 passed each. - jouleclaw-loader-gguf with fabric-meter: 299 passed, 2 ignored. - jouleclaw-harness-econ: 14. - mesh-meter with hw-meter-apple: 10. - l3-jouleclaw: 2 + 4, with 2 + 2 ignored. - energy_probe prints "ioreport : declined (CpuRailFrozen)". - ~/.jouleclaw/fabric-calibration.json is unchanged (2026-06-11 11:02:14). - cargo check --all-targets, with and without ioreport, on x86_64/aarch64-unknown-linux-gnu and x86_64/aarch64-apple-darwin: 0 warnings. - clippy --no-deps --all-targets --features ioreport shows only warnings that were already there: - the duplicated link attribute, now apple_ioreport.rs:112; - the b"IOReportChannels\0" literal in sum_energy_uj, now :823; - ledger.rs:631-634. - rustdoc --document-private-items shows only its three earlier warnings. Found, not fixed here: one batch refresh is two advances, so the rule can say Live on this frozen host - During the first mutation pass, the per-thread-cache mutant ran about a dozen real probes in one lib test process. Two of them, on two test threads, said Live, and the oracle tests failed: - "probe said Live (probed=CpuRailFrozen, cached=Live), the oracle did not see the CPU rail keep moving"; - "left: [CpuRailFrozen, ...] right: [Live, ...]". - CPU Energy moved in one batch refresh, 2,588,973,125 -> 2,643,843,378, between 17:54 and 17:59:38. That mutant ran from 17:58:24 to 17:58:34. - On this chip CPU Energy is the sum MCPU0 + MCPU1 + PCPU: 311,045,470 + 313,039,186 + 2,019,758,721 = 2,643,843,377, against 2,643,843,378. - A watcher sampled the Energy Model group every ~16 ms: 20 minutes with no refresh, then the next one caught one. The refresh landed as two steps of every cluster and of CPU Energy, 64 ms apart: t=1076.463s CPU Energy 2643843378 -> 2705607863 (+61764485) sample#56065 t=1076.528s CPU Energy 2705607863 -> 2705611298 (+3435) sample#56068 (MCPU0 +9470641 then +788, MCPU1 +9747542 then +844, PCPU +42546301 then +1802.) - The probe's intervals are ~31 ms: a 25 ms tick plus a ~6 ms sample. A probe whose window holds such a refresh sees the same rail advance in two distinct intervals, which is Live under the current rule. - A probe lands on a refresh rarely: its window is 1 s, and the two refreshes seen here came about 30-34 minutes apart. When it does, production caches Live for the whole process. The counter would then sum batch-refreshed CPU rails, plus the GPU, tagged ModelBased: the over-claim this branch exists to stop. - The same mutant on the final tree left the lib tests at 66/66, with no refresh during that run. - The rule itself (the same rail advancing in two distinct intervals) is the cross-crate rule every sibling crate shares, so it is left unchanged here. One direction would demand what a live rail shows and a refresh cannot: a rail that advances in most intervals over a minimum count, for example at least 5 intervals with 80% of them advancing. The M5 Max GPU rail changed in 166 of 176 samples ~28 ms apart. A refresh gives at most 2 or 3 advancing intervals in a row. Not done here - Changing the liveness rule, per the finding above. It needs one decision across jouleclaw-energy, eoc-meter, proof-cascade, sandbox-meter and jcp-energy. - The counter's own read() still samples every channel. It is declined on both Macs. - No CI job runs these ioreport tests: .github/workflows is off-limits to this branch. Co-Authored-By: Claude Opus 5.5 <[email protected]>The jouleclaw-energy verifier found the defect on 2026-09-30, on the M5 Max (macOS 27.0.1). The liveness rule said Live when the same CPU rail advanced in two separate sample intervals (LIVE_RAIL_ADVANCES = 2). That rule assumed a batch refresh lands as one step. It does not always. A watcher sampling the Energy Model group alone every ~16 ms saw `CPU Energy` step 2,643,843,378 -> 2,705,607,863 (+61,764,485), then 2,705,607,863 -> 2,705,611,298 (+3,435) 64 ms later. MCPU0, MCPU1 and PCPU each stepped twice too. The probe samples every ~31 ms, so a probe window that holds such a refresh sees two advancing intervals and says Live, and the status is cached for the process. That happened: two real probes said Live while the oracle saw no sustained movement, and both coincided with a refresh. Refreshes come ~30 minutes apart, so this is rare, but it is exactly the failure the rule exists to prevent. On this M5 Max `CPU Energy` has held at 2,705,611,298, that refresh's second step, in every run for this commit that printed it. The rule now asks for a sustained advance. A rail is live only if the same rail (same key: name + ordinal) meets three bars after some interval: - it advanced in at least MIN_LIVE_ADVANCES = 5 separate intervals; - those were at least 80% of every interval observed so far (advancing * 5 >= intervals * 4); - the samples observed so far span at least MIN_LIVE_WINDOW = 300 ms of wall time, measured from the first usable sample. It is evaluated after each interval, and Live is declared the first time all three hold. A burst of steps fails the window while it is young, and fails the share once it is not. The rule needs each sample's wall time, so RailWatch::observe now takes it: watch() reads its injected clock as each sample returns, and tests script that clock. The probe keeps sampling until the rail is Live, or until the budget (1 s) has elapsed and at least 5 intervals were observed, and never past 5 budgets. Otherwise the verdict is Frozen, Absent or Unavailable, as before. Nothing else changed: per-rail keys, consecutive-sample pairing, a decrease or a first appearance is not an advance, one advance per rail per interval, the group-only subscription, the thread-CPU tick, the pure status mapping, the OnceLock cache, the scripted leaves, try_detect_given / admits_counter, counts_once, and the "ioreport liveness: status=" line. The cost: a live rail is declared after 300 ms at the least, not ~60 ms. A frozen verdict still takes the ~1 s budget. The test oracle had the same weakness. It said CpuRailMoved(true) when one rail changed at two reads, so a refresh that coincided with a false Live vouched for it. Now it is CpuRailMoved(true) only if one rail changed at least 5 times, at no fewer than 80% of the read pairs it compared. It keeps reading past its window until it has compared 5 pairs (6 reads), so a live rail could meet the bar, and it stops at 5 windows. Too few pairs is still a test failure, never a verdict. The per-read tally is now a pure OracleTally, so the bar runs on scripted reads too. The docs that described the old two-step rule are updated: the constants, CpuRail, RailWatch, the macos_ioreport module comment, IoReportStatus::{CpuRailFrozen, Live}, CPU_RAIL_LIVENESS_BUDGET and ioreport_status. Tests. The rule tests use a scripted 31 ms interval unless noted. (*) marks the ones that must fail under the old ">= 2 advances" rule. That was checked by temporarily restoring the old rule: 13 tests failed, every (*) test among them. The F9 test failed on its two-step script: "a refresh in two steps (5 samples) must be frozen", left: Live. - (*) h_the_two_step_refresh_is_frozen: the observed +61,764,485 / +3,435 refresh is Frozen at every position tried. The steps land in intervals (3,4), (1,2), (1,3), (3,5), (20,21) and (31,33), beside a moving GPU rail and sub-rails that step with it. - (*) i_a_refresh_in_three_to_five_steps_at_the_start_is_frozen: 3, 4 or 5 steps within 150 ms at the start (30 ms a sample) are Frozen. At five steps every interval so far has advanced; only the window stops it. - (*) j_the_m3_ultra_two_steps_on_each_die_are_frozen: DIE_0 and DIE_1 each step twice within 64 ms, together or one interval apart, at the start and mid-probe, amid same-named per-die channels. Frozen. - d_a_rail_advancing_every_interval_is_live_at_five_advances_and_300_ms: not Live at 279 ms; Live at the 10th interval (310 ms), and the probe stops there. At 60 ms a sample it is Live at the 5th interval (exactly 300 ms); at 59 ms, only at the 6th. - k_a_live_rail_missing_one_interval_in_five_is_live_two_is_not_yet: missing 1 interval in every 5 is Live at 8 of 10. Missing 2 of the first 5 (100 ms a sample) is not Live at 3 of 5 or at 7 of 9; it is Live at 8 of 10. Missing 2 in every 5 is Frozen. - l_a_live_rail_on_a_loaded_host_is_live_after_five_intervals: at 100 ms intervals, Live after exactly 5 intervals (500 ms). - f_the_budget_stops_the_probe_only_once_five_intervals_are_seen: with fewer than 5 intervals the probe keeps sampling past the budget, and the 5-budget cap stops it (at 2 s ticks: 3 intervals, all advancing, Frozen). - sustained_needs_five_advances_80_percent_and_300_ms: each bar at its edge. - probe_with_the_two_step_refresh_is_frozen and probe_with_a_sustained_advance_is_live: the same rule through probe_with's fakes. The sustained case is Live at 12 intervals, 300 ms. - F9, the real host_status() on scripted IOReport leaves. The old "two advances -> Live at the third sample" script is now a sustained advance: Live after 6..=14 samples and >= 300 ms of real wall time. Also added: (*) the two-step refresh through host_status() gives CpuRailFrozen. - the_oracle_does_not_count_a_refresh_as_movement: the two-step refresh, a single step and a five-step burst are no movement; 5 of 5, 16 of 20 and every read are movement; 15 of 20 and 4 of 4 are not. - Minimums follow the rule. A frozen verdict holds at least 5 ticks, and the budget comments say five intervals. Mutations. Each was a temporary edit, then `git checkout`; the tree was clean after every one. All four were caught (tests that failed): - MIN_LIVE_ADVANCES = 2: f_..., l_..., sustained_... - 80% share check removed: a_decrease_is_not_an_advance (a rail rising and falling in turn), i_..., k_..., sustained_... - 300 ms window check removed: c_..., d_..., i_..., k_..., probe_with_a_sustained_advance_is_live, sustained_..., host_status_is_the_rule_on_the_production_composition - oracle back to ">= 2 changes": the_oracle_does_not_count_a_refresh_as_movement Evidence. - M5 Max, unloaded: eoc-meter has 45 lib tests (37 before) and the once-per-process test, all passing. `ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false (oracle: 7 reads over 3.062004833s with a helper thread burning CPU; changes per CPU rail [0], per GPU rail [5], of 6 compared pairs)` - M5 Max, 3 runs of all eoc-meter tests beside a helper process sampling every IOReport channel in a tight loop: 3/3 ok, 45 + 1 passed each. The oracle read past its 3 s window to reach 5 compared pairs: `ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false (oracle: 6 reads over 3.664727125s with a helper thread burning CPU; changes per CPU rail [0], per GPU rail [4], of 5 compared pairs)`, then 11 reads over 3.54 s, then 9 reads over 3.57 s. The fresh probe said Frozen every time (12-14 intervals over ~1 s). - M3 Ultra (10.10.10.2, macOS 27.0.1), 1 run of the lib and once-per-process test binaries: 45 + 1 passed. `ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false (oracle: 18 reads over 3.101584s with a helper thread burning CPU; changes per CPU rail [0, 0], per GPU rail [17], of 17 compared pairs)` - The whole eoc workspace: 621 passed, 0 failed, 3 ignored. The ignored three are the pre-existing ignored doctests. - clippy --no-deps on eoc-meter, all targets: no new warnings. The only two are the pre-existing ones (a duplicated #[link] attribute, and a b"...\0" literal). The aarch64-linux cross clippy could not run here: blake3's C build finds no Linux sysroot. Co-Authored-By: Claude Opus 5.5 <[email protected]>The CPU-rail liveness rule said Live once the same rail advanced in two separate sample intervals. On 2026-09-30 the jouleclaw-energy verifier, on the M5 Max under macOS 27.0.1, showed that two advances can be one batch refresh. A group-only watcher sampling every ~16 ms saw `CPU Energy` go 2,643,843,378 -> 2,705,607,863 (+61,764,485) and then, 64 ms later, -> 2,705,611,298 (+3,435); `MCPU0`, `MCPU1` and `PCPU` each stepped twice with it. This probe samples about every 31 ms, so a probe window that holds such a refresh sees two advancing intervals and says Live, cached for the process. Two real probes did exactly that while the independent oracle saw no sustained movement, coinciding with a refresh. Refreshes are ~30 minutes apart, so this is rare, but it is the failure the rule exists to prevent: a Live status selects IOReport for the life of the process, and every later refresh lands whole in whichever span straddles it, sealed as a measurement. The rule now asks whether one rail KEEPS advancing. Live iff the same rail (name + ordinal) advanced in at least MIN_LIVE_ADVANCES = 5 separate intervals, those advancing intervals are at least 80% of all intervals observed so far (advancing * 5 >= intervals * 4), and the samples so far span at least MIN_LIVE_WINDOW = 300 ms of wall time from the first sample. It is evaluated after every interval and latches Live as soon as all three hold. A refresh of fewer than five steps can never pass; one of five or more steps inside 150 ms cannot pass at any steady sampling pace (five advances in 150 ms need intervals of 37.5 ms or less, and then 300 ms holds eight intervals or more, of which the burst advanced in under two thirds). A live rail at the probe's idle cadence proves itself at the tenth interval, ~310 ms. The rule needs each interval's wall time, so RailWatch::observe takes the wall-clock reading with each sample, and the loop (`watch`) passes its clock read just after the sample returned; RailWatch records them in `at`. LIVE_RAIL_ADVANCES (2) becomes MIN_LIVE_ADVANCES (5), with MIN_LIVE_WINDOW and a pure `sustained(advances, intervals)` beside it. The loop keeps sampling until Live, or until the 1 s budget has elapsed AND at least five intervals were observed; the hard cap stays five budgets. Frozen, Absent and Unavailable map exactly as before. Unchanged: per-rail keys, reading each interval against the previous sample, a decrease or a first appearance not being an advance, one advance per rail per interval, the group-only subscription, the thread-CPU tick, the pure status mapping, the OnceLock cache, the F9 fault seam, the Span::stop and run_attributed rules, and the "ioreport liveness: status=" line. The test oracle's "moved" verdict had the same hole: CpuRailMoved(true) on two changes, so the refresh that fooled the probe could also vouch for it. It now says moved only if one rail changed at five reads or more, at four reads in five at least (oracle_saw_it_keep_moving, its own literals). It reads for its whole window and past it until it has five reads, never past four windows, and still fails the test on fewer. Under contention it did extend: two runs took 5 reads over 3.64 s and 3.19 s of a 3 s window. Tests added (scripted clock, ~31 ms an interval unless stated). Those marked (*) fail under the old two-advance rule, as the old-rule run below shows. - (*) the_observed_two_step_refresh_is_frozen: [a,a,a,a+61764485, a+61764485+3435], then flat for a 1 s budget, and the same with the steps two intervals apart: Frozen, never live at any interval. - (*) a_refresh_in_three_to_five_steps_at_the_start_is_frozen: three, four and five steps in consecutive intervals inside 150 ms: Frozen. - (*) an_m3_ultra_refresh_stepping_twice_per_die_is_frozen: DIE_0 and DIE_1 each stepping twice within 64 ms, in step and apart: Frozen. - a_rail_advancing_every_interval_is_live_not_before_five_advances_and_300_ms: live first at interval 10 at 31 ms, 5 at 60 ms, 5 at 75 ms (where 300 ms arrives at the fourth), and stays live. - a_live_rail_missing_one_interval_in_five_is_live (80%, wherever the miss falls: interval 10 at 31 ms, 6 at 100 ms) and a_rail_missing_two_intervals_in_five_is_not_yet_live (60% never, and a rail that missed two of its first five is live only at the tenth, 8 of 10). - a_loaded_host_with_a_live_rail_is_live_after_five_intervals: 100 ms intervals, live at the fifth (500 ms), not the third or fourth; the loop stops there with six samples and five ticks. - two_rails_that_advance_in_turn_are_not_one_live_rail, for two dies and for twin channels, so per-rail keying still decides a verdict now that two single steps no longer could. - Keep-sampling: the_probe_samples_until_live_or_long_enough_to_tell and the_probe_loop_needs_the_budget_and_five_intervals_to_stop (continues past the budget while fewer than five intervals, stops at five, stops at the cap with three); the loop hands the rule each sample with the clock just after it (the_probe_loop_stops_as_soon_as_a_rail_is_live pins `at`). - the_oracle_says_moved_only_for_a_rail_that_changed_at_four_reads_in_five. Real-host-composition (F9) tests, through host_status on a scripted IOReport: the "two advances -> Live" script is now a rail advancing at every sample (Live after 6..=14 samples, and a scripted real_run shows from its own record that it stopped at the first interval the rule allows); new the_production_status_of_the_observed_two_step_refresh_is_frozen (CpuRailFrozen, watched >= 900 ms of the budget); the two-dies test now has the dies advancing in turn for the whole budget, so a sampler that summed the dies would be called Live; frozen-status tests expect five intervals. The same in-turn change was made to the probe_with sum test. Mutations, each a temporary edit restored with git checkout (tree clean after each): - the old rule (MIN_LIVE_ADVANCES = 2, no fraction, no window): 17 failed, including all three (*) tests and the F9 two-step refresh ("a two-step refresh was called Live"). - MIN_LIVE_ADVANCES = 2: 6 failed (e.g. the loaded host went live at the third interval, Some(3)). - fraction check removed: 5 failed (five-step refresh live at interval 10; in-turn dies live; 60% rail live). - window check removed: 7 failed (five-step refresh live at interval 5). - oracle back to ">= 2 changes": 1 failed, the oracle verdict test. Runs. sandbox-meter lib 83 passed, integration 1 passed, on this M5 Max (macOS 27.0.1). Three runs of the lib and integration binaries beside an all-channel IOReport reader (~337 ms a sample): 0 of 6 failed, each "ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false". The M3 Ultra (macOS 27.0.1), same binary: 83 passed, "ioreport probe: probed=CpuRailFrozen cached=CpuRailFrozen oracle=CpuRailMoved(false) took=1.240480083s intervals=28 window=1.002887292s rails=[("DIE_0_CPU Energy", 0, 0), ("DIE_1_CPU Energy", 0, 0)]". Consumers: jcp-gateway + jcp-action-receipt 32 passed, joule-code-remediate 13 passed, joule-ui-interaction + joule-ui-serve 12 passed, the rest of sandbox-rs 186 passed. clippy --no-deps on sandbox-meter, all targets: no warnings on aarch64-apple-darwin or x86_64-unknown-linux-gnu (x86_64-apple-darwin shows one pre-existing unused-mut in detect_given, untouched here). Not done here, deliberately: - A refresh of five or more steps whose last interval is stalled well past its tick (a heavily loaded host) could still meet all three conditions; no such refresh has been seen, and the observed ones have two steps. - The same rule in the four sibling meters is being changed separately. Co-Authored-By: Claude Opus 5.5 <[email protected]>The liveness rule said Live once the same CPU rail advanced in two separate sample intervals. That was meant to stop a batch refresh (one step, then a hold) from vouching for a frozen rail. But a refresh is not always one step. On an M5 Max running macOS 27.0.1, the jouleclaw-energy verifier ran a group-only watcher that sampled the Energy Model group every ~16 ms. It saw `CPU Energy` go 2,643,843,378 -> 2,705,607,863 (+61,764,485), then 64 ms later -> 2,705,611,298 (+3,435). `MCPU0`, `MCPU1` and `PCPU` each stepped twice with it. The probe's intervals are ~31 ms, so a probe window that holds such a refresh sees the rail advance in two intervals and says Live. That answer is cached for the process. There, two real probes said Live while the oracle saw no sustained movement, at the time of a refresh. Refreshes come ~30 minutes apart, so this is rare. But it is exactly the failure the rule exists to prevent: the meter is selected over a frozen rail, and every later refresh lands whole in some bracket. The rule, the same in all five crates that carry it. Live iff, after some interval, the same CPU rail (name + ordinal): - advanced in at least MIN_LIVE_ADVANCES = 5 separate intervals; - advanced in at least 80% of all intervals seen so far (advances * 5 >= intervals * 4); - was watched for at least MIN_LIVE_WINDOW = 300 ms of wall clock since the first sample. The rule is checked after each interval. Live is declared, and the probe stops, the first time all three hold. The pure rule now takes each sample's wall-clock reading. The probe passes the read's `end`, and the tests script the clock. Sampling goes on until Live, or until the 1 s budget has passed and at least 5 intervals were seen. The hard cap stays at 5 budgets. On a quiet M5 Max a live rail is now called live after ~10 intervals (~310 ms), not 2. Unchanged: per-rail keys; consecutive samples; a decrease or a first appearance is not an advance; one advance per rail per interval; the group-only subscription; the thread-CPU tick; the pure status mapping; the OnceLock cache; the F9 fault hooks; proof-cli's messages; the RAPL per-zone wrap code; and the "ioreport liveness: status=" log line. The oracle. It called a rail moving after 2 changes, so a two-step refresh in its window could vouch for a false Live. Now it calls a rail moving only if the rail changed at 5 or more of its reads and at 80% or more of them. Its count moves into an `OracleTally` that the tests feed scripted reads. It still extends its window while it has fewer than 5 reads, but the cap is now 8 budgets (was 3). Under contention it needed up to 6.8 s of the old 9 s cap to get 5 reads. Too few reads is still a failure, never a "not moved". Tests. Those marked (*) fail under the old rule. - (*) The observed two-step refresh, with MCPU0/MCPU1/PCPU stepping beside it. Frozen after the whole budget: in the pure loop, in the composition, and in the real `host_status()` on scripted reads (CpuRailFrozen). - (*) A refresh in 3, 4 or 5 steps within its first 150 ms, back to back or spread out: Frozen. - (*) The M3 Ultra shape, where DIE_0 and DIE_1 each step twice within 64 ms: Frozen. - (*) Two advances of one rail over 300 ms: not live. Five advances in six intervals over 600 ms: live. - A rail that advances every 31 ms interval is Live at the tenth (310 ms). It is not Live after 5 advances (155 ms) or 9 (279 ms), and it stays Live. - A rail missing one interval in five is Live at exactly 80% (8 of 10). One missing two in five is never Live. One that missed two of its first five is not yet Live at intervals 7-9, where the count and window are already met, and is Live at the tenth. - On a loaded host (100 ms intervals), Live comes at the fifth interval (500 ms), not the third (300 ms). - The probe keeps sampling past the budget while it has fewer than 5 intervals and stops at 5. It stops at the cap with 3. - The oracle's bar, on scripted reads: a one- or two-step refresh is never "moved". Scripts whose meaning changed were rewritten so they still catch what they were written for: - "Two advances -> Live" became a sustained-advance script. It must be Live after 5 or more intervals and 300 ms or more of the real clock, at the first interval where the tests' own reading of the record says so. - The one-step-per-die scripts became two dies that advance in alternate intervals, so a sum of the rails advances every interval and fails. - The decrease and first-appearance tests now come after four advances over 400-500 ms. - The record test pins which wall reading the rule gets. `assert_verdict_matches_record` recomputes the new rule with the tests' own literals, and checks that a Live run stopped at the first interval that met it. Mutations. Each was a temporary edit, restored with git checkout; the tree was clean after each. - The old rule (2 advances, no share check, no window): 21 lib tests fail, including every (*) test, e.g. "a two-step refresh counted as a live rail: Run { verdict: Some(Live), intervals: 4, ...". - MIN_LIVE_ADVANCES = 2: 7 fail. For one, the loaded host is "Run { verdict: Some(Live), intervals: 3, samples: 4, took: 300ms". - Share check removed: 6 fail, e.g. "a rail advancing in 60% of intervals counted as live" and "a refresh in steps [1, 2, 3, 4, 5] counted as live: Run { verdict: Some(Live), intervals: 10, ...". - Window check removed: 8 fail, e.g. "a refresh in steps [1, 2, 3, 4, 5] counted as live: Run { verdict: Some(Live), intervals: 5, samples: 6, took: 150ms" and "Live after 127.469542ms of window". - Oracle back to ">= 2 changes": "two steps in consecutive reads: the oracle called a refresh moving". Runs. - proof-cascade tests, 3 times, beside an all-channel IOReport sampler (150-570 ms a sample on this loaded host): 68 + 1 + 1 passed each time, e.g. "status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false probe_intervals=29 ... oracle_reads=5". The sampler was killed after each run. - M3 Ultra (macOS 27.0.1): the lib test binary was built here, run once, then deleted. 68 passed, with "status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false probe_intervals=27 ... oracle_reads=20". - The proof-rs workspace is green, and proof-conformance prints "all checks passed". clippy --no-deps shows no new warnings; the two in proof-cascade were already there. Not done here: - The Live branch is tested only on scripted rails. Neither Mac exposes a live CPU rail to an unprivileged reader, so a real Live has not been seen. - The other four crates that carry this rule get it in their own commits. Co-Authored-By: Claude Opus 5.5 <[email protected]>The CPU-rail liveness rule said Live once the same CPU rail had advanced in two separate sample intervals. On 2026-09-30 the jouleclaw-energy verifier found a batch refresh that passes it. On this M5 Max (macOS 27.0.1), a group-only watcher sampling the Energy Model every ~16 ms saw a refresh published in two steps: CPU Energy 2643843378 -> 2705607863 (+61764485) CPU Energy 2705607863 -> 2705611298 (+3435) 64 ms later and MCPU0, MCPU1 and PCPU each stepped twice with it. The probe's intervals are ~31 ms (a 25 ms thread-CPU tick and a ~6 ms sample), so a probe window holding such a refresh sees two advancing intervals and says Live, and the Live is cached for the process. Two real probes did say Live while the oracle saw no sustained movement, coinciding with a refresh. Refreshes are ~30 minutes apart, so this is rare, but it is exactly the failure the rule exists to prevent: the rung selected on a frozen rail, and every later refresh billed whole to whichever metered span straddles it. The rail on this host still holds at 2,705,611,298, the value that refresh left. The rule is now, for one CPU rail (name + ordinal): - it advanced in at least MIN_LIVE_ADVANCES = 5 distinct intervals; - those are at least 80% of all the intervals observed so far (advancing * 5 >= intervals * 4); - the window observed so far spans at least MIN_LIVE_WINDOW = 300 ms of wall time from the first usable sample. It is checked after each interval, and the rail is Live from the first interval at which all three hold. The probe keeps sampling until Live, or until the 1 s budget has elapsed and at least five intervals were observed, capped at five budgets. A live rail is now called live about 300 ms in (12 intervals on an idle host) rather than ~60 ms. The rule needs each interval's wall time, so watch() reads the clock after every sample and hands that reading to RailWatch::observe; the tests script it. Everything else stands: per-rail keys, consecutive samples, a fall or a first appearance is no advance, one advance per rail per interval, the group-only subscription, the thread-CPU tick, the pure status mapping, the OnceLock, the test seams, and the "ioreport liveness: status=" line. The oracle gets the same spirit. CpuRailMoved(true) now needs one rail to have changed at 5 or more reads making up at least 80% of its reads. Its all-channel reads are 0.13 s or more apart, so even a refresh spread over 150 ms shows as three changes at most, and a refresh coinciding with the probe can no longer vouch for the Live it caused. It extends its window, up to 5x, until it has five reads after the first, and still fails as TooFewReads when it cannot. Its loop is now oracle_watch_with, over a read function and a clock, so its verdict is tested on scripts. Tests, on a scripted clock at 31 ms an interval unless stated; (*) fail under the two-advance rule: - (*) the observed two-step refresh, with the steps in consecutive intervals, 62 ms apart, and on a host at 160 ms an interval: Frozen; - (*) a refresh in 3, 4 or 5 steps within 150 ms at the start of the window, and in 5 steps mid-window: Frozen; - (*) the M3 Ultra shape, both dies stepping twice within 64 ms: Frozen; - a rail advancing in every interval: Live at the 10th (310 ms), not at the 5th (155 ms), on either die; - missing 1 interval in 5 (exactly 80%): Live; missing 2 in 5: never; on a loaded host, 2 early misses: not at 5 of 7, 6 of 8 or 7 of 9, Live at 8 of 10; - a loaded host at 100 ms an interval: Live at the 5th (500 ms), not at the 3rd, where the window is already met; - keep-sampling: at 600 ms an interval the probe goes past the budget to five intervals (where a live rail is proven Live, at 3 s); at 1.5 s it stops at the 5 s cap with four; - through probe_with: a sustained advance is Live at 10 intervals, and (*) the two-step refresh is Frozen; - through host_status itself (the real opener, burn tick and clocks, with scripted FFI reads): (*) the two-step refresh is CpuRailFrozen; a sustained advance is Live with 5+ advances over a 300+ ms window by both the probe's clock and the tests' own, stopping as soon as it is live; - the two-die summing check now has the dies advance in turn (each in half the intervals, summed in all): two one-step dies, summed, are only two advances, which no longer trips the rule; - the oracle on scripts: the two-step refresh is CpuRailMoved(false) (it was true), a three-step refresh false, a change at every read true at the 5th, 4 in 5 true, 3 in 5 false; slow reads are extended to five, reads too slow for that are TooFewReads. Mutations, each a temporary edit restored with git checkout, the tree clean after each: - MIN_LIVE_ADVANCES = 2: 4 tests fail (the two-step refresh on a loaded host, loaded-host Live before the 5th interval, keep-sampling, the stopping rule); - the 80% check removed: 4 fail (the 80% test, the 5-step refresh, two dies in turn through probe_with and through host_status); - the 300 ms window removed: 6 fail (Live at the 10th, the 80% test, the 5-step refresh at the start, the sustained advance through probe_with and through host_status, the stopping rule); - the oracle's bar back to >= 2 changes: both oracle tests fail. The whole old oracle (2 changes, 2 reads) fails with "a two-step refresh vouched for a moving rail: OracleRun { verdict: CpuRailMoved(true), reads: 4, ..., cpu_changes: [2] ..."; - the whole old rule (2 advances, no share, no window): 15 fail, every (*) test among them. Runs. The crate's tests pass (51 lib, 1 integration). Three runs under a helper process sampling IOReport over all channels back to back (~250-280 ms a sample, load average 14-16), 51/51 each: ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false (oracle: 6 reads over 3.857275292s of a 3s window with a helper thread burning CPU; changes per CPU rail [0], per GPU rail [4]) ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false (oracle: 7 reads over 3.051596791s of a 3s window with a helper thread burning CPU; changes per CPU rail [0], per GPU rail [6]) ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false (oracle: 9 reads over 3.133878417s of a 3s window with a helper thread burning CPU; changes per CPU rail [0], per GPU rail [5]) On the M3 Ultra (macOS 27.0.1), the same binary, 51/51: ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false (oracle: 21 reads over 3.033503042s of a 3s window with a helper thread burning CPU; changes per CPU rail [0, 0], per GPU rail [19]) jcp-cli (it builds; it has no tests), jcp-conformance (with the conformance binary: all checks passed, C9 energy provenance honesty among them) and jcp-intercept pass. Clippy --no-deps and rustdoc over private items with -D warnings are clean on jcp-energy. Co-Authored-By: Claude Opus 5.5 <[email protected]>The CPU-rail liveness rule said Live once one CPU rail had advanced in two separate sample intervals. A macOS 27 batch refresh can meet that on its own. On this M5 Max (macOS 27.0.1, 2026-09-30) a watcher sampling the Energy Model group alone every ~16 ms saw one refresh land in two steps: CPU Energy 2643843378 -> 2705607863 (+61764485) CPU Energy 2705607863 -> 2705611298 (+3435) 64 ms later with MCPU0, MCPU1 and PCPU each stepping twice alongside. The probe's intervals are ~31 ms (a 25 ms CPU tick and a ~6 ms group sample), so a probe window that holds such a refresh sees two advancing intervals and says Live, and that answer is cached for the process. It was seen to happen: under the per-thread-cache mutant, two real probes said Live while the oracle saw no sustained movement, at the moment of a refresh. The rule is now the family rule for the five crates that probe this rail (this commit changes this crate only): - Live iff one CPU rail (same key: name + ordinal) advanced in at least MIN_LIVE_ADVANCES = 5 distinct intervals, those are at least four in five of all the intervals observed so far (advancing * 5 >= intervals * 4), and the samples so far span at least MIN_LIVE_WINDOW = 300 ms of wall-clock time from the first. It is evaluated after every interval, and is Live as soon as all three hold. - The probe keeps sampling until Live, or until the 1 s budget has elapsed and at least 5 intervals were observed; hard cap 5 budgets. - The rule is handed each sample's wall-clock time: `observe(sample, at)`, where the loop passes its clock's reading as each sample comes back, and the tests script it. To be Live, a refresh would have to step in eight of the probe's first ten ~31 ms intervals, which is what a live rail does and a refresh does not. A live rail (the M5 Max GPU rail changed in 13,125 of 13,614 samples 100 ms apart, 96%) is Live after ten unloaded intervals (310 ms), or five 100 ms intervals on a loaded host (500 ms). Everything else is unchanged: per-rail keys, consecutive samples, a decrease or a first appearance is no advance, one advance per rail per interval, the group-only subscription, the thread-CPU tick, the pure verdict-to-status mapping, the OnceLock cache, REAL_RUNS and GROUP_OPENS, the F9 seam (which also records the time handed with each sample now), new_given/admits, counts_once, the consumers, and the "ioreport liveness: status=" line. The docs that described the two-interval rule (module doc, RailWatch, IoReportStatus, ioreport_status, detect_meter, the Apple estimator) describe this one. The test oracle is held to the same bar. It said CpuRailMoved(true) at two changes, so a refresh landing in its window could vouch for a false Live. Now it says so only if one rail changed at five or more of its reads and at four in five or more of them, over its whole window. It reads on past its 3 s window until it has five reads (capped at four windows), and a CPU rail read fewer times still fails the calling test as no evidence. The freshness test's "cannot tell fresh from stale" disclosure is now bounded: a run whose probe said Live must show the oracle seeing a CPU rail change inside the probe's window (Live means the rule was handed one advancing in five or more intervals over 300 ms or more), so the disclosure can only cover runs that were not Live, and a stale sampler can never make one Live. Tests. Those marked (*) fail under the old two-advance rule. - (*) the_observed_two_step_refresh_is_frozen: the refresh above, with its steps in consecutive intervals and 64 ms apart. - (*) a_refresh_in_three_to_five_quick_steps_is_frozen: 3, 4 and 5 steps in the first 150 ms, one per interval. - (*) an_m3_ultra_two_die_two_step_refresh_is_frozen: DIE_0 and DIE_1 each stepping twice within 64 ms. A modelled shape: the M3 Ultra has been seen to refresh in one step per die. - (*) the_production_status_of_a_two_step_refresh_is_frozen: the same refresh through the real uncached host status function (F9 seam). - a_rail_advancing_every_interval_is_live_not_before_five_advances_and_300_ms: Live at interval 10 unloaded (310 ms) and 5 loaded (500 ms); not at 9 (279 ms) or at 3 (300 ms). - a_rail_missing_one_interval_in_five_is_live (Live at exactly 8 of 10) and a_rail_missing_two_intervals_in_five_is_not_yet_live. - the_window_runs_from_the_first_sample_on_the_clock_handed_in, and two_rails_advancing_in_turn_are_not_one_live_rail. - The watch-loop tests at 31 and 100 ms intervals and past the budget: the probe stops at five intervals, keeps going with fewer, stops at the cap. - the_production_status_of_a_rail_that_keeps_advancing_is_live replaces "two advances -> Live after exactly 3 sample calls": Live after 6 to 13 sample calls, each sample's time handed to the rule inside the tests' own clock bracket, at least 300 ms watched, and not Live one sample earlier. - the_oracle_says_moved_only_for_a_rail_that_kept_changing, and the_oracle_reads_on_past_its_window_until_a_live_rail_could_meet_its_bar. The lib tests go from 66 to 76. Mutations, each a temporary edit restored with git checkout (the tree clean after each), lib tests with --features ioreport: - the old rule (two advances, nothing else): 13 fail, all four (*) among them - MIN_LIVE_ADVANCES = 2: 6 fail - the four-in-five check removed: 3 fail - the 300 ms window check removed: 6 fail - the oracle back to ">= 2 changes": 2 fail - also caught: the window measured from the clock's origin (3 fail), the loop handing the rule a zero time (3), ">" for ">=" in the share (2). Runs. jouleclaw-energy lib and integration tests with and without --features ioreport (76 + 1 + 1, and 24 + 1 + 1). Three times beside an all-channel IOReport sampler looping in another process, all green: ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false oracle_reads=7 ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false oracle_reads=5 ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false oracle_reads=5 The oracle extended itself to five reads in up to 5.5 s where its 3 s window gave fewer. On the M3 Ultra (macOS 27.0.1), 76 passed: ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false oracle_reads=16 The scripted sustained rail went Live after 13 sample calls there and in every M5 Max run (rule window 302-322 ms). The consumers pass: jouledesk-github (12), jouleclaw-loader-gguf --features fabric-meter (367), jouleclaw-harness-econ (14), mesh-meter --features hw-meter-apple (10), l3-jouleclaw-binding (6). clippy --no-deps --features ioreport gives the same six warnings as before, none new; rustdoc adds none. Not done here: - The freshness test's Live assertion never runs on either Mac here, since both are CpuRailFrozen; it guards a live host. - The budget rule stops at five intervals once 1 s has passed, even if the window is still under 300 ms. That only happens if the first sample took more than ~700 ms, and it declines a live rail, which is the safe side. Co-Authored-By: Claude Opus 5.5 <[email protected]>The owner settled the grade question this branch left open: a vendor energy counter (RAPL, NVML, Apple IOReport) gives a real reading; what it lacks is independent provenance, not accuracy. So a figure read from a counter that is live for the domain it claims is HwShunt. proof-cli already tagged a --measure figure that way, but the crate still said elsewhere that a macOS reading was ModelBased, and the CLI's own comment called the grade "an open grade decision, not settled here". This commit makes the specification, the schema and the docs agree with the code, and adds the liveness requirement where the grades are defined. No behaviour changes; one test is tightened. What changed: - SPEC.md section 5 and proof-schema's Provenance::HwShunt doc. HwShunt now names vendor energy counters next to a shunt, a power meter and the SMC. Both carry the family's liveness MUST word for word: a vendor counter's reading MAY be graded as a hardware reading only while the counter is demonstrably live for the domain the figure claims, observed advancing under load (for Apple IOReport, the CPU energy rail itself, in at least 5 sample intervals, in at least 4 of every 5, over at least 300 ms, while the probing thread burns CPU). A counter that is present but not live, frozen or republished in lumps, MUST NOT produce a hardware-graded figure. The spec adds that the fallback here is ModelBased or Estimator. Energy::measured's doc points at the requirement. No conformance vector is touched. - proof-cascade meter.rs. The module header said "a macOS reading is ModelBased, not a true coulomb-shunt HwShunt; Linux RAPL is the shunt-grade path". It now says a figure from this meter is HwShunt on both platforms, states the liveness condition, and says what each platform does about it. The docs on measure_marginal_uj and power_over say the same, and power_over's doc gives the probe's thresholds. The Linux section is new and honest about a gap: there is no liveness probe for RAPL. It is selected whenever a zone is readable, a frozen counter gives Some(0), which is not a figure, and nothing tells a counter that tracks the load from one that drops a lump into the busy window. - meter.rs's RAPL comment said energy_uj is "world-readable on most distros". It has not been since late 2020: kernel commit 949dd0104c49 ("powercap: restrict energy meter to root access", 5.10, backported to stable) made it root-only as the CVE-2020-8694 / PLATYPUS mitigation. An unprivileged process usually reads no zone, so the meter declines unless it runs as root or read access has been granted. - proof-cli main.rs. The module doc called T0 and T1 "both real with measured energy". Their energy is proof-solvers' wall time x a ~15 W power model, tagged ModelBased. It also said "T2/T3 reference stand-ins", but the CLI runs resident_z3_ladder, which has no T2/T3 rung. Those three goals come back unproved, as `proof` prints ("proved 6/9 ... T2=0 T3=0"). Both are fixed, and the --measure comment now states the grade. - proof/README.md. It said `--measure` gives an HwShunt reading "where an energy interface is present". It now says where the meter produces one, which on Apple Silicon needs a CPU rail seen to be live. It adds that on the Macs measured so far (macOS 27.0.1, M5 Max and M3 Ultra) --measure declines and every figure stays ModelBased, and that RAPL is root-only on Linux. It no longer says the solver rungs have "measured energy". It no longer says clippy is warning-free: that holds for the Linux target, but on macOS proof-cascade's IOReport FFI has two lints. (The "via macmon" text was already gone inbedef9b0.) - proof-cli tests/measure.rs.bedef9b0had already moved this test off a second probe in the test process and onto the decision the CLI prints. But the printed status was tied to the CLI's behaviour for only two of its four values. A CLI that printed ioreport_status=CpuRailAbsent while acting on CpuRailFrozen passed: run against the unmodified test at906ed1a6, it gave "ok. 1 passed". Now every IOReport reason in the output must belong to the printed status. A declined meter must print its own status's reason. The status must be one of the four. Off macOS it must be Unavailable, with no IOReport reason. Verified on the M5 Max (macOS 27.0.1, z3 present). The whole proof-rs workspace passes: proof-cascade 68, ioreport_once 1, ioreport_unopenable 1, measure 1, conformance pack 4 + soundness 5 + lib 1, receipt 8, schema 6, solvers 11. proof-conformance prints "all checks passed". The measure test passed 3 runs, each "declined=true available=false ioreport_status=CpuRailFrozen". clippy --no-deps --tests shows only the two existing proof-cascade lints, meter.rs's duplicated #[link] attribute and its hand-built nul-terminated string. It is clean for x86_64-unknown-linux-gnu, which type-checks the test's off-macOS branch but does not run it. rustdoc reports no warnings. Falsified, each mutation restored from a saved copy and its hash checked afterwards: - the CLI prints IoReportStatus::CpuRailAbsent in place of the status it used: "the CLI printed ioreport_status=CpuRailAbsent but gave the reason for CpuRailFrozen"; - prints Unavailable: "... printed ioreport_status=Unavailable but gave the reason for CpuRailFrozen"; - prints Live: "on macOS the meter is available exactly when IOReport is Live"; - prints !available: "the CLI's output disagrees with the decision it printed"; - the frozen reason is replaced by a generic one: "the meter declined with ioreport_status=CpuRailFrozen, but not for that status's reason". Not done here: - a RAPL liveness probe. Under the new MUST, the Linux path's only evidence is the bracket. Co-Authored-By: Claude Opus 5.5 <[email protected]>An allowed `tools/call` comes back with `_meta["science.transaction/jcp"]`: the sealed receipt, `measured_uj` and `budget_remaining_uj`. Since this branch made the gateway seal each call's real provenance, `measured_uj` is the call's cost, which is a hardware reading only where the span was one. On a host with no live energy counter (every macOS 27 host checked) it is the meter's estimate, and the plaintext said nothing about which: the grade existed only inside `receipt_b64`, so a client that read the number without decoding the COSE envelope took an estimate for a measurement. The key keeps its name, because clients already read it. Beside it the block now carries `energy_provenance` (`HwShunt`, `ModelBased` or `Estimator`) and, when the figure reads only part of the call (a GPU counter with no CPU record), `energy_coverage: "Partial"`: the same two labels, with the same omit-when-Whole rule, that the sealed receipt carries. `attach_receipt` takes the cost `Energy` instead of its bare `u64`, so the label cannot drift from the figure. The change is additive: nothing is removed from the block, the receipt bytes are unchanged, and no published JCP or AR-1 vector moves. AR-1's `joules_uj` was documented as "Measured cost in microjoules". It is the work's cost however it was obtained, graded by `energy_provenance`; the field doc, the family table ("measured joules") and AR-1-spec §2.2 now say so, and §9's energy cross-check says it holds only where the figure is `HwShunt`. Doc-only: the encoding and the pack are untouched. Tests. `the_caller_gets_its_receipt_back` now decodes the receipt and asserts that the plaintext `energy_provenance` and `energy_coverage` equal what it sealed, and that a non-hardware meter's call is labelled `Estimator`; on this host it observes `"energy_provenance":"Estimator","measured_uj":26577`. A new `the_plaintext_figure_carries_its_grade` covers every cost shape: Estimator, HwShunt whole, HwShunt Partial, ModelBased. Falsified: removing the `energy_provenance` insert fails both tests (`energy_provenance beside measured_uj`; `left: None, right: Some("Estimator")`). jcp-gateway 13/13, jcp-action-receipt 17 + 3 conformance pass, and `jcp-conformance` reports all checks passed. Clippy is clean on jcp-gateway; its one jcp-action-receipt warning is in the untouched `ar-verify` binary. Co-Authored-By: Claude Opus 5.5 <[email protected]>The crate description promised "a JouleContract receipt with hardware-measured joules sealed by a sandbox meter attestation". The crate does not do that on a host without a live CPU energy counter (every macOS 27 host checked): there its receipts carry an estimate, sealed as JCR-1 `Estimator` / JouleContract `Unaccounted`, as its own module doc says. The description now says the joules are graded by how they were obtained: a hardware reading apportioned to the run and anchored to a sealed meter envelope only where the host exposes a live CPU counter, an estimate sealed as one everywhere else. Two `FormalEntry`/`FormalOutcome` docs that called the proof cascade's `ModelBased` accounting "measured joules" are corrected too. `a_proven_formal_obligation_lets_the_fix_ship` and `a_refuted_formal_obligation_reverts_the_fix` returned before asserting anything when `z3` was absent, so on a solverless host (the joule-code-rs CI job installs none) they passed in 0.00 s whatever the formal gate did. They now follow the convention proof-solvers and joulecontract-proof already use, and that the standards-build workflow's "Install z3" step documents: the suite passes with or without a solver and asserts something either way. With `z3` the solver's verdict is asserted as before. Without it, both assert the gate fails closed: the obligation is Unproved (neither proved nor refuted), the `formal-proof` gate fails while every other gate passes, the fix is reverted and not shipped, and the discharge evidence is still written. Each logs which branch it asserted. Where a solver is supposed to be installed, `JOULE_FIX_EXPECT_Z3=1` turns a missing `z3` into a test failure instead of the solverless branch; this is documented on the test helper and in Cargo.toml beside the proof dependencies. No workflow is changed, so CI still runs the solverless branch until a job installs z3 and sets the variable. The formal gate's detail, which a failed gate copies into its receipt entry, and joule-fix's summary line both said "real z3" even with no solver on PATH. They now say "no z3 on PATH: SMT goals left undecided" in that case. Verified, 13/13 each way: with z3 on PATH; with z3 removed from PATH; with no git identity (GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 and user.useConfigOnly=true); and with no identity and no z3. `JOULE_FIX_EXPECT_Z3=1` with no z3 fails both tests ("JOULE_FIX_EXPECT_Z3=1, but there is no working `z3` on PATH"); with z3 present it passes. Falsified: letting an unproved obligation pass the gate (`all_proved` without `unproved == 0`) fails both tests with no z3 on PATH. The same mutation against the previous tests passes both, in 0.00 s. Clippy --no-deps on the crate is clean. Co-Authored-By: Claude Opus 5.5 <[email protected]>`PassportBuilder::build` accepted any pairing of `measurement_method` with `compliance.eu_ai_act.inference_energy_measured`. That includes `IoReport`, which `MeasurementMethod::provenance` itself grades `Estimator`, paired with `inference_energy_measured: true`. A sealed passport could then disclose estimator energy as measured inference energy under the EU AI Act field. It also accepted `Estimated` with `true`. The rationale beside `IoReport` was "estimator until the IOReport FFI ships". The FFI shipped long ago. The real reason, after the previous commits: a vendor energy counter is a real reading, and what decides its grade is provenance. Was the counter live for the domain the figure claims? A bare `IoReport` records no such observation, and where the CPU rail is frozen (macOS 27.0.1, for an unprivileged reader) the host's figure is the estimator's. So `IoReport` stays `Estimator`, and the stale wording is replaced in the variant doc, the crate doc and the test. The fix. - New `MeasurementMethod::IoReportLive`: IOReport read while its CPU energy rail was observed live (`IoReportStatus::Live`). It is graded `HwShunt`, like RAPL and NVML. Display: "IOReport (live)". It is additive, so existing serialized passports still deserialize. - `is_live_hardware_reading()` is true iff the method's grade is `HwShunt`. - `build()` returns `PassportError::MeasuredWithoutHardwareReading { method, provenance }` when `inference_energy_measured` is true and the method is not a live hardware reading. That means a bare `IoReport` or `Estimated`. Unclaimed, every method still builds. Tests: `a_measured_claim_needs_a_live_hardware_reading`. `IoReport` and `Estimated` with a measured claim are refused with their method and grade, and build without it. `IoReportLive`, `Rapl`, `Nvml` and `NvmlRapl` build with the claim. The message names the method, the grade and the rule. The mapping test covers `IoReportLive` → `HwShunt`, and the display test covers its label. jouleclaw-compliance: 53 pass. Falsified (restored with `git checkout`, tree clean): a gate that refuses only `Estimated`, and so accepts `IoReport` with a measured claim, fails the test with "IOReport paired with a measured claim was not refused". 52 / 1. Co-Authored-By: Claude Opus 5.5 <[email protected]>The headline said, on one run, "energy 375387 uj vs 450000000 uj frontier-only → 1198.762× less for the same verified output". The 450 J baseline charged a frontier model for all nine goals; the cascade proved six. The three it did not prove produced nothing, so the comparison was not for the same output: it credited the cascade with 150 J of frontier work it never replaced. `EnergySummary` gains `proved_baseline_uj`, the frontier-only baseline summed over the proved goals alone, and `proved_ratio_milli()`, that baseline over EVERY joule the cascade spent, the goals it failed on included, in thousandths rounded down (`None` when nothing was spent; `Some(0)` when nothing was proved). Both sides are charged conservatively: the frontier run only for the output it would have to match, the cascade for its failures too. The existing `baseline_uj` and `energy_ratio_milli()` are unchanged (the conformance check C3 and the receipt wire use them), and their docs now say they charge every goal. The CLI now prints, verbatim from one run (the T1 figures are wall-time models of a resident z3 and vary several-fold between runs): proved 6/9 · tiers T0=4 T1=2 T2=0 T3=0 energy 58892 uj for all 9 goals vs 300000000 uj frontier-only for the 6 proved → 5094.070× less for the same verified output (frontier-only for all 9 goals: 450000000 uj, 7641.105× — but 3 of them were not proved here, so not the same output) The all-goals figure stays, with its goal count, and is never called the same output; with nothing proved, or nothing spent, the CLI says so instead of printing a ratio. Tests: - `the_same_output_ratio_charges_the_baseline_only_for_proved_goals` (cascade): three goals, two proved, with the tests' own literals (T0 50 uJ, T1 5 mJ, T3 50 J): the same-output baseline is 2 × 50 J, the ratio is exactly 100,000,000,000 / 10,150 = 9,852,216 thousandths, below the all-goals ratio; a cache hit is a proved goal for zero joules; nothing spent gives no ratio, nothing proved gives zero. - `tests/headline.rs` (CLI): runs `proof` and checks the printed figures against the per-goal rows of the same run — the total is the sum of the rows, the same-output baseline is (proved rows) × 50 J, the all-goals one is (rows) × 50 J with the unproved count, both ratios are the printed baselines × 1000 ÷ the total, rounded down, and "same verified output" is claimed exactly once, on the proved-goals line. Falsified: printing the all-goals baseline and ratio on the same-output line fails the CLI test ("the same-output baseline is 6 x 50 J"); charging the proved baseline for every goal fails both tests. Each restored through git. Co-Authored-By: Claude Opus 5.5 <[email protected]>On both Macs that run this suite (macOS 27.0.1, an M5 Max and an M3 Ultra) IOReport's CPU energy rail is frozen for an unprivileged reader, so the meter has correctly declined and `proof --measure` has never printed a figure there. The kernel keeps a live counter of its own: `proc_pid_rusage(getpid(), RUSAGE_INFO_V6)`'s `ri_energy_nj`, the cumulative CPU energy it attributes to this process's threads, in nanojoules. It is public libproc API and needs no privilege. On the M5 Max it changes every ~0.5-0.7 ms under load, holds near 1-2 mJ/s idle, and keeps the energy of threads that have exited; on both Macs the probe below calls it Live. Design. `measure_marginal` (new; `measure_marginal_uj` is now its figure) runs `work` on every spare core and meters it with the first live source, in order: on macOS this process's counter, then IOReport; on Linux RAPL. For the process counter there is no bracket and no idle subtraction: the counter is read before the workers start and after every one has been joined, and the figure is its advance ÷ every call that completed, so it covers exactly the calls counted, and other processes are excluded by construction. What the figure claims, `MeterSource::domain`, says so: this process's CPU energy as the kernel attributes it to its threads — not other processes, not child processes, not GPU, ANE or DRAM — and everything this process burns in the window, so a caller must not run other work in the process while it meters. It is graded HwShunt (`MeterSource::provenance`; the owner's decision: a kernel reading of the CPU's own energy counters), and `Measurement::energy()` builds the schema reading from it. The counter is used only while it is live, by the same sustained-advance rule as IOReport and RAPL: through the same probe loop, it must advance in at least five sample intervals, in at least 80% of them, over at least 300 ms, while the probe burns 25 ms of thread CPU before each re-read. `process_energy_status` probes once per process and caches it; `meter_source` admits only `Live`, and the process rung has its own gate on the cached status besides. The bracket paths keep their own gates whatever ranks above them: `power_over` now gates on IOReport's own status and `rapl_power_over` on RAPL's, not on the overall selection, so a live process counter can never unlock a frozen IOReport rail. IOReport is not probed by the selection when the process counter is live. `proof --measure` prints the decision it acted on (`source=`, `process_energy_status=`, `ioreport_status=`, `rapl_status=`), why each source not used was not used, the figure with its source and provenance, and its domain. On this M5 Max, verbatim: --measure: hardware meter available=true · source=ProcessEnergy · process_energy_status=Live · ioreport_status=CpuRailFrozen · rapl_status=Unavailable measured: 20819 uj / proof · provenance HwShunt (measured_uj present, the schema's strongest tier) · source ProcessEnergy receipt: closed at T0:automation · 215 uj · provenance ModelBased · spec Ungrounded · verified true The receipt is unchanged: still sealed from `first`, the cascade's own ModelBased figure for the first goal. The CLI's old line "the standard's headline is now a real hardware reading" is gone: the per-goal figures stay ModelBased, and it now says so. Tests. Hardware tests check the meter against an independent oracle — the tests' own binding of `proc_pid_rusage`, their own `getpid`, a raw 464-byte buffer read at byte 336, their own loop and literals — and every branch asserts: - the FFI pin: three production reads each lie between two oracle reads, and advance across 50 ms burns; the struct is 464 bytes with `ri_energy_nj` at 336, the macOS 27.0 SDK header's numbers; - `tests/process_energy.rs` (serialized): the probe is Live exactly when the oracle sees the counter keep moving, and then it is selected first; a fixed cheap closure meters as a non-zero HwShunt figure three times within 3x; a closure that sleeps 2 ms meters at under a tenth of one that computes for 2 ms, also while four busy child processes burn CPU beside it; - `tests/process_energy_frozen.rs` drives the whole chain with the counter held (`simulate_frozen_process_energy`, a hook that can only turn the meter off): Frozen, never selected, no figure, the work never runs, one probe run; - `proof --measure` (proof-cli test): the source is the first live one in order, every printed reason is its printed status's own, the receipt equals the first goal's row and stays ModelBased, and where the test process's own counter keeps moving the CLI must have printed a non-zero HwShunt ProcessEnergy figure with its domain; - pure, on every platform: the probe composition on scripted counters (live at the twelfth 25 ms interval; constant, one-, two- and four-step counters Frozen; unreadable Unavailable), the measurement's arithmetic and read order on a scripted counter each call raises by exactly 7,000 nJ (exactly 7 uJ; it holds only if the counter is read before the first call and after the join), rounding, a falling or unreadable counter giving no figure and an unreadable one never running the work, the macOS order (IOReport not even asked when the process counter is live), and the grading and domain of every source. The IOReport tests now assert on IOReport's own gate (`bracket_admitted`, `bracket_marginal_uj`), and `ioreport_once.rs` / `ioreport_unopenable.rs` cover both probes (each runs once per process; with no Energy Model group and the counter held, every meter declines). Measured, verbatim from the test logs (debug builds, macOS 27.0.1): - M5 Max: `process liveness: status=Live source=Some(ProcessEnergy) oracle=Oracle { moved: true, reads: 35, changes: 35, advanced_nj: 1061927078 }`; cheap closure, three calls: 355, 355, 360 uJ; compute 2 ms = 2732 uJ, sleep 2 ms = 7 uJ, sleep 2 ms beside 4 busy children = 6 uJ. - M3 Ultra (the lib and process_energy test binaries and `proof`, built from this commit's code before its last doc-comment and README edits, copied to /tmp on 10.10.10.2, run once with --nocapture, deleted): `process liveness: status=Live source=Some(ProcessEnergy) oracle=Oracle { moved: true, reads: 29, changes: 29, advanced_nj: 5293659397 }`; cheap closure 671, 676, 685 uJ; compute 2 ms = 6520 uJ, sleep 2 ms = 2 uJ, beside 4 busy children = 9 uJ; `measured: 44537 uj / proof · provenance HwShunt · source ProcessEnergy`; 88/88 lib tests and 3/3 process_energy tests passed; IOReport there `CpuRailFrozen` as before. Falsified, each restored through git before the next (proof-cascade and proof-cli test suites): - reading the parent's pid: the FFI pin fails ("round 1: the production read 177161340 is outside the oracle's [897071438, 897071438]"), and so do the liveness test ("the probe says Frozen; the oracle saw the counter change at 35 of 35 reads"), both metering tests and the CLI test ("this test's own process counter changed at 24 of 24 reads, yet the CLI metered with none"); - a constant counter: the same five tests ("the production read 123456789 is outside the oracle's [...]"); - liveness ignored at the selection (the counter chosen whenever it reads): the frozen-chain test ("a held process counter was selected"), the unopenable test and the pure ordering test fail; - the probe run but its verdict discarded (always Live): the frozen-chain and unopenable tests fail ("a held counter was called Live"). `cargo test --workspace` and the proof-conformance binary pass; `cargo check --workspace --tests` and `cargo clippy --workspace --all-targets` for x86_64-unknown-linux-gnu are clean; clippy on macOS reports only the two IOReport FFI lints the README already lists. Not done: the counter's accuracy is the kernel's; nothing here checks it against an external meter. Its domain is CPU only, so GPU-bound work would meter low, and a caller that runs other threads in the process while it meters is charged for them. The process figure resolves 1 uJ per call; `Some(0)` from it means under half a microjoule, and the CLI says so. Co-Authored-By: Claude Opus 5.5 <[email protected]>The per-process rung graded every span that advanced `ri_energy_nj` as `HwShunt`, `Coverage::Whole`. That count covers only this process's threads. A child's energy is its own, and the kernel never adds it to the parent's count, not even when the child is reaped. So a closure that drives a child process was under-counted while its figure claimed the whole span. SPEC §5 forbids that: a figure must not claim coverage of a domain its counter does not read. The previous commit named this under "Not done here" and said grading it `Partial` would need the caller's help. It does not: the same `rusage_info_v6` shows it, and so does the kernel's list of children. The rule, in `process_span`: a span whose count advanced is whole only when no child process can have worked in it unseen. That means two things. The process's reaped children's CPU time (`ri_child_user_time` + `ri_child_system_time`) held across the span, and at the span's end `proc_listchildpids` lists no child, running or exited and not yet reaped. Otherwise the span is `Energy::measured_partial`: `HwShunt`, `Partial`, the same rise. A failed child count is `Partial` too, since nothing then shows the figure whole. A child the process had before the span counts as well, because the meter cannot tell whether the span's work drove it. `measure` lists the children before it reads the count, so a child reaped between the two shows up in one or the other. A span whose count did not advance is still the model's whole-task figure. What the kernel does, measured on an M5 Max, macOS 27.0.1, with the C compiler and small C and Rust programs rather than assumed: - `ri_child_user_time` and `ri_child_system_time` are at bytes 96 and 104 (`offsetof` against the 27 SDK). They are in Mach absolute time units: a `yes` that ran ~190 ms showed 4,565,473 of its own, and reaping it raised the parent's by 4,565,861. `ri_energy_nj` did not move on the reap. - Spawning a child by absolute path, or letting it run, never moved those times; only the reap did (24 of 24 trials, posix_spawn and fork+exec alike). - `proc_listchildpids` returns a count of pids when given a buffer. It lists a killed child until it is reaped. Without a buffer it returns a size estimate (1,098 with no child), so the reader always passes one. - A failed `posix_spawn` can move the reaped children's time too: a spawn of a missing path moved it in 2 of 200 attempts (90-127 units). A program spawned by name is looked up by trying each `PATH` directory. Here, with 12 ahead of `/usr/bin`, `Command::new("yes")` moved the time at spawn in 7 of 24 spawns, and `/usr/bin/yes` by absolute path in none of 12. Such a span is `Partial` on that alone, which errs on the side of claiming less. It is documented on `process_span`. A run of the new left-running test, which then spawned `yes` by name, caught it: "reaped children's CPU 0 -> 115 Mach units" with nothing reaped. The tests now spawn `/usr/bin/yes`. The decoder reads the two child-time words next to the energy words. `process_energy::child_count()` wraps `proc_listchildpids` with a 64-pid buffer. Docs: the crate's ladder entry, the module, `Meter::ProcessEnergy`, `process_span`, SPEC §5.1 item 3 and the README's jcp-energy bullet now say that a span that reaps a child or ends with one is `Partial`. Tests. jcp-energy has 80 lib tests (78 before) and its integration test. All pass with and without `--features nvml`, and three runs in a row were green: - Layout: the child-time offsets equal the C compiler's 96 and 104 and `offset_of!` on the transcribed struct. Decoding reads all four words and nothing else; on a literal buffer, 446,368 + 4,119,493 = 4,565,861. - Pure, on every platform: on a count that rose 0.48 J, a reaped child, a child left at the end, both, and a failed child count are each `HwShunt`/`Partial` with the same 480,001 µJ. Neither is whole. A held count is the model's whole figure whatever the children did. - Hardware, each in a fresh process. A span that runs `/usr/bin/yes > /dev/null` for 200 ms and reaps it is `Partial`. The tests' own reads show that the reaped time rose and no child is left, so the reap alone decided it. A span that leaves `yes` running is `Partial`; the child is listed and the reaped time held, so the left child alone decided it. A pure 200 ms in-process burn, with no child before or after, is whole. The old `a_child_process_work_is_not_attributed` is now the reap test, and keeps its check that the child's energy is not in the figure. - The tests' oracles are their own: `proc_listchildpids` through their own declaration, counting nonzero pids in a zeroed buffer, and the child times through the transcribed struct. A guard kills and reaps every child, so a failing test leaves none running. Falsified, each by a temporary edit restored with git checkout (git diff was empty after each): - Ignoring children (`whole = true`) fails the pure test and both hardware child tests: "a span that reaped a child claimed to cover it: Energy { measured_uj: Some(1368), ..., coverage: Whole }" and "a span that left a child running claimed to cover it: ... coverage: Whole". - Ignoring the reaped children's time fails the pure test and the reap test only. - Ignoring the children left at the end fails the pure test and the left-running test only. - A `child_count` that always says 0 fails the left-running test. jcp-intercept (12) and jcp-conformance (1 unit test and 5 pack tests) pass. The jcp-conformance binary prints "all checks passed", C9 included, and jcp-cli runs and exits 0. clippy --no-deps -D warnings is clean on jcp-energy for aarch64-apple-darwin (with and without `nvml`), x86_64-apple-darwin and x86_64-unknown-linux-gnu, and `cargo doc` with `-D warnings` is clean. M5 Max, verbatim: process energy: a span running `yes > /dev/null` for 200 ms, then reaping it -> 409 µJ (0.000409 J), HwShunt, coverage Partial; the child's own count 229352810 nJ (0.229 J); reaped children's CPU 0 -> 4673857 Mach units; children left [] process energy: a span leaving `yes > /dev/null` running -> 29048 µJ (0.029048 J), HwShunt, coverage Partial; children at its end [41504] (the child 41504); reaped children's CPU 0 -> 0 Mach units M3 Ultra, verbatim. The lib test binary was built here with `target-cpu=apple-m1`, copied to 10.10.10.2 (sha256 060761dd2b0e2efc... on both ends), run with --nocapture, then deleted. 80 of 80 passed: process energy: a span running `yes > /dev/null` for 200 ms, then reaping it -> 1322 µJ (0.001322 J), HwShunt, coverage Partial; the child's own count 1245873027 nJ (1.246 J); reaped children's CPU 0 -> 5041160 Mach units; children left [] process energy: a span leaving `yes > /dev/null` running -> 137541 µJ (0.137541 J), HwShunt, coverage Partial; children at its end [30980] (the child 30980); reaped children's CPU 0 -> 0 Mach units process energy: 200 ms of one thread's CPU -> 1366000 µJ (1.366 J), HwShunt, coverage Whole ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false Not done here: - Work handed to the GPU, the Neural Engine, or another process over IPC (a daemon serving this one) is still not in the figure and is not detected. rusage has no per-process GPU energy. - A grandchild that outlives the child that started it is reparented to launchd and is not listed. The reap of its parent still makes that span `Partial`, but a later span cannot see it. - A long-lived child the process keeps makes every span `Partial` while it lives. That is safe, but coarse. Rebased onto25bfdcd9(the RAPL figure sums the package zones by name), which adds four RAPL lib tests and changes no line of this commit; the counts above, the M3 Ultra run's included, predate it. After the rebase jcp-energy has 84 lib tests (82 before this commit), all passing on the M5 Max with and without `--features nvml`, as do jcp-intercept, jcp-conformance (its tests and the binary) and jcp-cli. Co-Authored-By: Claude Opus 5.5 <[email protected]>On macOS 27, `proc_pid_rusage(getpid(), RUSAGE_INFO_V6)` (libproc: public, no privilege, no entitlement) returns `ri_energy_nj`, the cumulative CPU energy the kernel attributes to this process's threads. Unlike IOReport's CPU rails, it is live for an unprivileged reader: it advanced in 20 of 20 intervals here, under 25 ms of CPU each. It is the first live CPU energy counter eoc-meter has on a Mac. Its domain is the trap, and the design follows from it: this process's threads' CPU energy, all of them, and nothing else. Not a child process (a child's energy is its own, and is not added to the parent's when reaped), not the GPU, the Neural Engine or DRAM. eoc's counter brackets stages that may run in a subprocess (eoc-tools' shell tool, HumanEval's python) or on the GPU (mlx, mlc, llama.cpp's default offload). A span over such work would read only this process's share and be tagged measured: a figure for work the counter did not see. So: - `MacosProcessEnergyCounter` (`JouleCounter`, cumulative µJ = `ri_energy_nj` / 1000, name `process-energy`) is constructed only where it is live. Its doc states the domain exactly. - `process_energy_status()` probes it once per process and caches the answer, by the same sustained-advance rule and `probe_counter` the RAPL and NVML rungs use, burning this thread's CPU between samples (that burn is this process's work, so a live counter advances in every interval). It is `Unavailable` off macOS. - `detect()` does **not** select it, under any status of any rung. A caller whose metered work runs on the CPU in this process opts in with `detect_in_process()` (this counter where live, otherwise `detect()`) or `MacosProcessEnergyCounter::try_new()`. - A span the counter did not advance over is still no measurement; the consumers enforce that (eoc-cascade's `live_span`, eoc-streaming, eoc-local). The FFI is hand-rolled: `struct rusage_info_v6` field for field from the macOS 27 SDK, with compile-time asserts that it is 464 bytes and `ri_energy_nj` is at byte 336, as a C program compiled against the SDK prints. It is the crate's third scoped `allow(unsafe_code)` module. Tests. In the library, on scripted reads: the probe calls a counter that advances every read live; a constant, one step, or a failing `proc_pid_rusage` is declined. Only `Live` constructs the counter, which then reads this pid. A pid that does not exist reads nothing. `detect_with` never returns the process counter for any of the 36 combinations of NVML, RAPL and IOReport status, and `detect()` never does here; `detect_in_process` picks it exactly for `Live`, on macOS. Against the hardware, in an integration test binary of its own (`tests/ process_energy.rs`), because the counter's domain is the whole process and other tests in the library binary burn CPU on other threads; its tests take one lock. Each figure is checked against an oracle the test reads itself: `proc_pid_rusage` into a raw buffer at the SDK's offsets. No test can pass without asserting. - live: the status agrees with the oracle (live if it advanced in 80% of 20 intervals), and on Apple Silicon both must say live; `detect_in_process()` picks it; `detect()` does not. - 200 ms of one thread's CPU reads 0.2-20 J, inside the oracle's bracket. The burn is 200 ms of the thread's own CPU, by its CPU clock, at user-interactive QoS. A 200 ms wall-time spin failed on this M5 Max under a load average near 70 ("174844 µJ for 200 ms of one thread's CPU work": it got 90 ms of CPU). And the bound is for a performance core: 200 ms of CPU on an M3 Ultra efficiency core (background QoS) reads 0.013 J, against 0.72 J at default QoS and 1.10 J at user-interactive (a C probe, not this test); - 500 ms asleep reads under 0.1 J; - a child process (`yes` to /dev/null, run at least 300 ms and until its own counter reads 0.5 J) is not attributed: the parent's counter, read before the spawn and after the reap, moves under 0.1 J and under a fifth of the child's own, while the reaped child's CPU time shows it worked. Measured, verbatim from the test logs: M5 Max (macOS 27.0.1, load average ~50): process-energy oracle: 20 of 20 intervals advanced status=Live oracle_live=true detect_in_process()=process-energy detect()=powermetrics one-thread burn of 200.001792ms thread CPU in 205.349583ms wall read 600514 µJ = 0.601 J (3.00 J per CPU-second) 500 ms asleep read 512 µJ = 0.0005 J child ran 315.771709ms, its own counter read Some(700069134) nJ, its CPU (reaped) 289.941208ms; the parent's counter moved 1708 µJ = 0.0017 J (three runs before it: burns of 0.483, 0.444, 0.408 J; asleep 4894, 2735, 47 µJ; the parent moved 14311, 1404, 1199 µJ while a child read 0.54-0.61 J) M3 Ultra (macOS 27.0.1, test binaries built for apple-m3, copied, run, deleted): process-energy oracle: 20 of 20 intervals advanced status=Live oracle_live=true detect_in_process()=process-energy detect()=powermetrics one-thread burn of 200.003584ms thread CPU in 200.008542ms wall read 1329909 µJ = 1.330 J (6.65 J per CPU-second) 500 ms asleep read 184 µJ = 0.0002 J child ran 356.917167ms, its own counter read Some(1725566789) nJ, its CPU (reaped) 356.596542ms; the parent's counter moved 187 µJ = 0.0002 J So the counter is live on both chips. The first M3 Ultra run failed `a_child_processs_work_is_not_attributed` with "the child did not work: 120.042625ms of CPU": the child was stopped as soon as its counter read 0.5 J, which the M3 Ultra's `yes` reached in 120 ms (0.716 J), under the 200 ms CPU floor the test then demanded. A flaw in the test, not the counter; the parent had moved 843 µJ. The child now runs at least 300 ms, and the floor is 150 ms. The library's 62 tests passed there too. Falsified, each mutant restored byte-for-byte (sha256 checked), run against the library and the hardware binary: - the process counter as a `detect()` rung fails 3 tests (`detect_with( Unavailable, Unavailable, Unavailable)` picks it; `detect()` picks it); - a constant reading (counter only) fails the burn test ("0 µJ for 200 ms of one thread's CPU work"); a constant at the source makes the probe decline, and fails 4; - reading the parent's pid fails 2 (the burn is not the parent's); probing the parent's pid fails 4; - the constructor ignoring its status fails 2. clippy raises no new warning. `cargo check --tests` for x86_64-unknown-linux-gnu, with and without `cuda`, is clean. Co-Authored-By: Claude Opus 5.5 <[email protected]>joule-ui-serve's action handlers and the joule-ui-interaction examples' handlers are in-process work, verified rather than assumed: `Host::handle` calls the handler inside `run_interaction`, on the worker thread that holds the host lock, and nothing in joule-ui-serve or the examples spawns a process. So they now meter with `PowerMeter::detect_in_process()`, and on a host whose per-process CPU energy counter is live (both Apple Silicon hosts checked, macOS 27.0.1, where the machine-wide CPU rail is frozen) each interaction is sealed as a `HwShunt` reading of the energy the kernel attributes to this process's threads over the interaction, instead of the 20 W wall-clock estimate. Where the counter is not live the meter is `detect()`'s, as before. The figure is this process's, not one thread's: in joule-ui-serve it includes any worker that served a socket during the span. joule-ui-serve: `main` serves with `served_power_meter()`, which is `detect_in_process()`, and the end-to-end test now builds its meter from that same function and asserts, where the counter is live, that the meter is the process probe and that each ~24 ms served interaction is `HwShunt`. The examples: `real_meter` and `jcp_metered` use `detect_in_process()` and say which counter they read. With real readings an interaction of ~150 ms costs a fraction of a joule, not the estimate's 3 J, so six fires no longer spend `jcp_metered`'s 15 J grant; it now fires until the real JCP runtime denies one (at most 200), printing the first five and the last three, so the budget binds whatever the figures are. The grant itself is unchanged. SPEC §5 names the per-process counter among the reference meter's sources; the two Cargo comments say which meter each crate uses. Tests: joule-ui-serve, -interaction, -conformance, -mcp, -core, -a2ui and -render pass (72). The served interactions on this M5 Max: `Energy { measured_uj: Some(25908), ... provenance: HwShunt }` and `Some(23193)`. `gen_vectors` regenerated into a scratch directory is byte-identical to `conformance/v1/` (`diff -r` clean). The examples, release build, here: real_meter: render #1: allow · 195798 µJ [HwShunt] ... sealed chain: 3 links verified, 584156 µJ total — hardware counter deltas (HwShunt), this process's CPU energy over each interaction jcp_metered: 82 interactions sealed: 81 allowed; 15001229 µJ sealed, hardware counter deltas (HwShunt) from process-energy; 15000000 of the 15000000 µJ budget debited; 1 denied by the real JCP runtime once the budget was spent. (The per-interaction figure varied from 0.04 to 0.21 J: the handlers burn for 150 ms of wall time, and other builds were loading this machine.) Falsified, restored from git and re-run green: `served_power_meter` returning `detect()` fails `the_served_meter_never_seals_zero_or_an_estimate_as_a_reading`. Rebased onto7cf1f036with the commit before it; this commit's files are unchanged. joule-ui-interaction, -serve, -conformance, -mcp, -core, -a2ui and -render pass, and `gen_vectors` regenerated into a scratch directory is still byte-identical to `conformance/v1/`. Rebased again, ontoa16e36cf, with the commit before it; this commit's files are unchanged. The seven joule-ui crates pass (72), and `gen_vectors` regenerated into a scratch directory is still byte-identical to `conformance/v1/`. Co-Authored-By: Claude Opus 5.5 <[email protected]>219a7790851d85821aa8