jouleclaw-energy: IOReport only when its CPU rail keeps moving; consumers never zero-fill or relabel #117
Loading…
Reference in a new issue
No description provided.
Delete branch "jouleclaw-energy/cpu-rail-liveness"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
jouleclaw-energy's
detect_meter()pickedAppleIoReportCounterwhenever an IOReport subscription opened. Its consumers then turned a missing or zero figure into a number.What was wrong: on main
before=0 after=0.fabric_calibrate/mpp_energywould write~/.jouleclaw/fabric-calibration.jsonwith CPU members at about 0 µJ/GFLOP, labelled Measured (NaN ratio on the M3 Ultra). The fabric placer would then route matmuls to the CPU.tune_shape_livereturned 0 µJ when it could not measure, and a recorded 0 can never be replaced.uj: 0.What this does
AppleIoReportCounteris gated on the family rule. A declined counter hasis_available() == false,status()gives the reason, and everyread()isErr(NoCounter).detect_meter()falls through toAppleSiliconMeter, which is Estimator end to end.marginal_energymodule. The tuner records nothing unless every window was measured. The calibration examples refuse before writing and print the IOReport status. A pre-existing temp-file collision intier_storetests is fixed.Verification of this PR (head
962d34a7)--features fabric-meter--features hw-meter-apple~/.jouleclaw/fabric-calibration.jsonis unchanged.Later rounds on this branch
33f0b891,f9a91c2c,9acf777dandea61169c:67cd4e7b: RAPL and NVML are gated on the liveness rule. NVML's fallback power integral is ModelBased, as its SPEC always said.55c82eff: mesh-meter never grades a GPU-only figure HwShunt. mesh/1 has no coverage field, so it becomes an estimate of the request.f521fb17: l3-jouleclaw grades a fabric-priced projection by its weakest input.c7a4d710: IOReport is live only if every CPU rail it sums is live.c07d4c5b: RAPL sums everypackage-Nzone, on every socket, never onlyintel-rapl:0. Each zone wraps at its own range.Why: what IOReport actually does on macOS 27.0.1
Measured by reading channels directly on the M5 Max (dev) and the M3 Ultra (10.10.10.2), both on macOS 27.0.1:
CPU EnergyandDIE_<n>_CPU Energydo not move under load for seconds to minutes. On the M3 Ultra both dies held for at least 22 minutes, then jumped about 1.3 kJ each.+61,764,485 mJ, then+3,435 mJ64 ms later.A meter that sums rails therefore reports CPU work as GPU-only energy (M5 Max) or 0 (M3 Ultra), labelled as a reading. A liveness check that accepts one advance, or two, gives a false Live whenever its window holds a refresh. After that, each later ~1.3 kJ lump is booked whole to whichever span straddles it.
The rule: identical in all five backends (#110, #114, #115, #116, #117)
CPU Energy, or eachDIE_<n>_CPU Energy), keyed by name and ordinal:namereadspackage-N, on every socket, and each must be live. It never sumspsys, which already includes the packages, nordramor the sub-zones.max_energy_range_uj. A wrap whose range is unknown gives no figure; it never becomes a ~1.8×10¹⁹ µJ reading.Livedeclines the backend, so the crate's existing honest fallback runs. The reason is public:IoReportStatus,ioreport_status(),rapl_status(),nvml_status().admits(status)(Live only).How it was verified
Six rounds of implementation, each checked by independent adversarial verification.
Pure tests on scripted samples and a scripted clock. They cover:
Fault hooks inside the real IOReport opener and sampler. The tests drive the real production status through each script: opener fails, a single lump, a two-step lump, constant values, and a sustained advance. So a bypass written into the glue fails a test.
Hardware tests against an independent oracle. The oracle has its own subscription, reads absolute values, burns on a helper thread, and counts a rail as moving only on a sustained change. The tests check:
Mutation-tested. Each of the following was applied as a temporary edit and failed a test:
admits()that accepts CpuRailAbsent.The counts per crate are below.
Robustness. Each crate passes under artificial CPU load and while another process samples every IOReport channel in a tight loop. The latter is what made an earlier wall-clock pin flaky, so the job now runs on the shared CI host without that flake. Each also passes on the M3 Ultra.
CI: these tests are behind
cfg(target_os = "macos"), and this workspace runs on a linux runner. They reach CI through #102, which merges last. #102 runs every backend's lib and integration tests on macOS, and fails any step whose frozen-rail test did not printioreport liveness:.Merge order: #110, #114, #115, #116 and #117 are independent of each other. #102 goes after all of them.
Open questions, now decided and implemented (per "fix it all")
measured_ujthat carry estimates. Their grade now sits beside them in plaintext: jcp-gateway's_metagetsenergy_provenance, plusenergy_coveragewhen partial. The keys stay, for compatibility.Meterdefaults to Estimator, and the conformance fixture keeps HwShunt explicitly, so the vectors are byte-identical.Authorizer::chargecarries the figure's provenance.ri_energy_nj: unprivileged, live on both Macs, and attributed to the work. It is wired in as stacked PRs #120–#124, only where the work runs in-process. (Privilegedpowermetricsis live too, but machine-wide and root-only, so it is not used.)Still open
joule-ui-servegives it honest grades, and the per-process probe in #121 gives it real readings.🤖 Generated with Claude Code
The hardware tests for the CPU-rail liveness probe run in CI on the macOS runner, which is this M5 Max, and it is often heavily loaded: load averages of 24-104 have been seen. Round 3 left tests that only held on a quiet host, a cache that was pinned per thread rather than per process, and rail values nobody checked. This commit fixes those. The probe's tick - The tick now burns 25 ms of the probing thread's own CPU time, measured on CLOCK_THREAD_CPUTIME_ID. It gives up after 500 ms of wall time. - The old tick spun on the wall clock. On a loaded host the scheduler gave it only part of each 25 ms. With 36 extra spinners on this Mac, that old tick left intervals with only 8.6-11.8 ms of thread CPU. - The tick is still passed into watch(), so the scripted tests keep their scripted ticks. What the hardware tests assert now - The probe burns CPU. In every interval it counts, the probing thread must use at least half a tick (12.5 ms) of CPU time. Only the 500 ms wall cap could break that, and the assertion message says so. The probe records its thread CPU at each sample for this check. - The probe subscribes to the Energy Model group only. The median wall time of the probe's own sample calls must be under 50 ms. A group sample takes about 6-12 ms and an all-channel sample 130-800 ms. The probe records each call's duration, and the test prints the median. - The interval-count assertions are gone: ">= 10 intervals" and ">= 2 intervals plus the full budget". What stays is "a frozen or absent verdict took at least the 1 s budget". It is a lower bound, and load can only make the probe slower. No hardware test asserts a ratio of CPU time to wall time. - New test: the_probe_sampler_reads_each_cpu_rail_at_its_own_value. It checks the values the sampler reads, against an oracle on the oracle's own all-channel subscription. - The oracle reads every Energy Model CPU rail by name, once before and once after one probe sample. - If the oracle saw a rail hold, the probe's value must equal it. If the rail moved, the probe's value must lie between the two reads. - Before this, a sampler that returned constant 0, or gave each CPU rail the sum of all CPU rails, passed every test. The oracle - It keeps reading past its 3 s window, up to 4 windows, until it has at least 2 reads after its baseline. - With fewer reads it fails the test, with the read count and the reason. It never returns CpuRailMoved(false) on 0 or 1 reads. - The "ioreport liveness:" line now ends with oracle_reads=N. The test that prints it keeps its name. - Under load the oracle needed the extension: one run got 2 reads in 3.76 s. The pure rule: each case has its own test - A rail that goes down between two samples (a reset or wrap) has not advanced. - A rail missing from the previous sample only sets its baseline. - No verdict maps to Unavailable. - A probe that cannot subscribe to the Energy Model group, or whose subscription never returns a sample, maps to Unavailable through status_of_run(). It is never Live and never CpuRailAbsent. That last rule changes behaviour. A host where the group cannot be subscribed but the all-channel subscription opens used to report CpuRailAbsent. It now reports Unavailable, because the probe saw nothing. IoReportStatus's documentation says this. The enum and its variants are unchanged. The once-per-process cache - A process-wide AtomicUsize, HOST_PROBE_RUNS (built only for tests), counts the runs of the real host probe. The thread-local counter is removed. - the_liveness_status_is_probed_once_and_cached now starts 6 threads through a barrier. Each calls ioreport_status() twice, builds two counters and calls detect_meter. The test asserts the host probe ran exactly once in the process. - This corrects073fb7d8, which said the cache was pinned "once per process". Its thread-local counter only pinned "once on this thread": a per-thread cache passed all 50 tests. Evidence - M5 Max (macOS 27.0.1), clean test binary, 10 runs under 36 background spinners (2 x hw.ncpu). Load average went from 34.8 to 142.8 during the runs. The spinners were killed afterwards, and none was left. - Result: 10/10, "test result: ok. 55 passed". - Every run: "ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false oracle_reads=3..7". - Group probe: 5-10 intervals, median sample cost 7.16-10.24 ms. - Least thread CPU in any interval: 31.72-32.39 ms. - Oracle: 2-8 reads after the baseline. - M3 Ultra (27.0.1), the same binary: 55 passed. - Median sample cost 6.75 ms over 33 calls. - Rails DIE_0_CPU Energy and DIE_1_CPU Energy. - Sampler values equal to the oracle's: 28237793 and 30845042. - Oracle: 16 reads. Mutations. Each was a temporary edit, restored with git checkout, and git status was clean afterwards. - A decrease counts as an advance: 2 tests fail, "a fall counted as an advance". - A rail new in a sample counts as an advance: 1 fails, "a new rail counted as an advance". - No group subscription maps to Live: 1 fails. - Per-thread cache: 1 fails, "the host liveness probe ran 13 times in this process". With no cache at all, 41 times. - The sampler returns constant 0: 1 fails, "held at 1863990848 ... but the probe's sampler read 0". - Each CPU rail gets the sum of all CPU rails: - On the M3 Ultra it fails, "("DIE_0_CPU Energy", 0) held at 28237793 ... but the probe's sampler read 59082835". - On the M5 Max it is an equivalent mutant, because that group has one CPU rail. Summing every channel named *CPU* into it does fail there: "read 5567863491". - The tick sleeps: fails, "interval 0 of 29 burned only 6.086583ms". A no-op tick fails the same way: "interval 0 of 114 ... 8.550958ms". - The probe on the all-channel subscription: fails, "median sample call took 538.447416ms". - The oracle concludes after 1 read: 2 tests fail with the read-count message. - The mutations whose old assertions were removed are still caught: - watch() stopping after 1 interval fails 4 tests. - A 10x-fast probe clock fails, "decided CpuRailFrozen after 211.462541ms, before its 1s budget". - PROBE_TICK = 250 ms fails the scripted test (a). - The round-3 wall-clock tick passes on a quiet host. Under the same load it fails 3/3 on the new CPU pin. Other checks - No features: 24 passed. - Linux x86_64 and aarch64, with and without ioreport: cargo check passes. - Clippy: no new warnings. - Consumers: - jouledesk-github: 20/20. - jouledesk: 18 + 4. - loader-gguf with rayon and fabric-meter: 302 passed. Runs with only fabric-meter hit a tier_store temp-file race (NotFound) twice in 12 runs, in a file this branch does not touch; the 10 other runs passed. - harness-econ: 14. - mesh-meter with hw-meter-apple: 10. mesh-core: 3. - l3-jouleclaw: 2 + 4. Not done here - The counter's own read() still samples every channel. It is declined on both Macs today. - No CI job runs these tests: .github/workflows is off-limits to this branch. Co-Authored-By: Claude Opus 5.5 <[email protected]>Round 4 left one medium finding and four mutations that no test caught. This commit closes all five. The hardware tests run in CI on this M5 Max, which is often heavily loaded, so every new pin reads the probe's own record or a count. None of them is a wall-clock ratio. The once-per-process pin counts the real probe - PROBE_RUNS (test builds only) is now incremented inside probe(), the function that opens the Energy Model subscription and samples it. Round 4 counted in probe_status, the wrapper that the cache called. So a counter constructor that called probe() directly, uncached, passed all 55 tests (O6). - probe_status is gone; host_status caches probe() itself. - Tests that run the probe on purpose, uncached, call the same probe() inside deliberately(). That is a test-only scope which marks one call, on one thread, as not the host's run. It exists only in test builds, so no production path can use it and still build. - the_liveness_status_is_probed_once_and_cached still releases 6 threads through a barrier. Each thread calls ioreport_status() twice, builds two counters and calls detect_meter. The real probe must have run exactly once in the process. A constructor that probes uncached now fails it with "the real liveness probe ran 26 times". The tick is pinned from the probe's own record - probe_cpu_rail records each tick: the probing thread's CPU time, on its own CLOCK_THREAD_CPUTIME_ID, and the wall time. Every tick must show a whole PROBE_TICK (25 ms) of thread CPU or have hit the 500 ms wall cap, and there must be one tick before every re-sample. - Round 4 pinned only the CPU used per interval, from the end of one sample to the end of the next. That includes the sample call's own 6-8 ms, so a tick that burned 25% or 30% of PROBE_TICK passed under load (O1, O1b). Both now fail at any load, for example: "tick 0 of 66 burned 6.251291ms ... a tick burns 25ms of thread CPU time". - The record is checked against the tests' own thread clock, which is Mach thread_info, not clock_gettime. The thread must have used at least as much CPU as the probe says its ticks and its counted intervals used. A probe whose "thread CPU" clock was really the wall clock passes this only when the thread was almost never off the CPU. Under load it fails: "its ticks used 804.630251ms ... the thread used only 566.975ms". The no-subscription path goes through the pure mapping, and a test runs it - probe() is now probe_with(open, probe_on), given the real opener and the real probe. probe_with maps "nothing opened" through status_of_run to Unavailable, the same as a probe that saw no interval. - A test-only switch, NO_ENERGY_MODEL_GROUP, makes open_energy_model return None on one thread. a_probe_that_cannot_subscribe_to_the_group_is_unavailable runs the real probe() that way, on a host whose group opens, and asserts Unavailable with nothing probed. It also calls probe_with with an opener that opens nothing, and with a subscription that never returns a sample. - Round 4's O2 (probe() maps a failed subscription straight to CpuRailAbsent) passed on both Macs. It now fails with "the real probe with no Energy Model subscription ... left: CpuRailAbsent". The same bypass returning Live fails too, placed in probe() or in probe_with. The sampler is pinned to read afresh - New test: the_probe_samples_afresh_across_its_window. While the real probe runs, an oracle thread reads the Energy Model group every 10 ms and records when each read ran. It uses its own subscription, opened with the raw FFI, not open_energy_model. The probe records each sample call's start, end and values (test builds only). - For every probe sample with an oracle read wholly before it and one wholly after it: a GPU Energy rail that held must read the oracle's value, and one that moved must read between the two oracle reads. - If the oracle saw a GPU Energy rail change between two reads strictly inside the probe's window, the probe's own samples must show it changing too. - Fewer than two oracle reads inside the window is no evidence, and fails. - the_probe_sampler_reads_each_cpu_rail_at_its_own_value now holds GPU Energy to the same bracketing rule as the CPU rails, over two rounds on one subscription. - Round 4's O3 (the sampler returns its first sample's values forever) passed on both Macs. It now fails. So do a sampler that is stale per probe run and one that is stale per subscription. Example: "probe sample 1: ("GPU Energy", 0) moved 123056146417043 -> 123056172426143 across the oracle's reads either side of it, but the probe's sample read 123056142245337, outside them: not a fresh sample". - What this test can show depends on the GPU: - The CPU rails are frozen on both Macs, so they never can. - On this M5 Max, GPU Energy changes every ~28 ms. It moved inside the window in every run here. - An M3 Ultra's GPU moves only sporadically: in 113 of 11,716 100 ms samples over 25 minutes one day, and in about a fifth of 30 ms intervals today. Today the M3 Ultra caught all three stale samplers. - In a window where the GPU does not move, the test prints "no GPU rail moved inside the window: this run cannot tell a fresh sampler from a stale one", and a stale sampler would pass that run. Docs: the budget's doc no longer says "each still under a full tick of burn", and the all-channel sample cost is now quoted as 0.1-1 s. Evidence, on this exact tree - M5 Max, macOS 27.0.1, 10 runs under 36 background spinners (2 x hw.ncpu), 1-minute load average 23-57: 10/10 "test result: ok. 57 passed". All spinners were killed afterwards and none was left. Every run showed: - "ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false", with oracle_reads=8-13. - "the real probe ran 1 time(s)". - Probe: 7-12 intervals, median sample cost 6.98-7.79 ms, least tick 25.0005 ms of thread CPU, least interval 31.34 ms. - Freshness: the oracle saw GPU Energy change 43-52 times inside the window, and the probe's own samples changed in every interval (6-13 changes). - M5 Max, 5 more runs under 72 spinners (4 x hw.ncpu), load up to 75: 5/5. - M3 Ultra (10.10.10.2), the same binary: 57 passed. DIE_0 and DIE_1 read equal to the oracle (35656018 and 38872869). GPU Energy moved 9 times inside the probe's window, and the probe's samples 9 times. - 31 mutations. Each was applied, built, run, and restored with git checkout, with git status clean after each one. - These fail at least one test: baseline never moves; same-named rails counted under one key; a decrease, or a first appearance, counted as an advance; no subscription mapped to Live (in status_of_run, in probe_with, in probe); no subscription mapped to CpuRailAbsent (in probe_with, in probe); per-thread cache; no cache; constructor probing uncached; constant-0 sampler; three stale samplers; tick sleeps; tick is a no-op (the burn call, or the whole tick); tick burns 30% or 25%; tick burns on a helper thread; all-channel subscription; liveness decided on the sum of every Energy Model rail, or on the GPU rail; advances summed over rails; a declined counter reading 0; the fallback metering 0. - Two pass on a quiet M5 Max, and each is caught where it differs. The first gives each CPU rail the sum of all CPU rails. It is equivalent on the M5 Max's single rail, and on the M3 Ultra it fails: "DIE_0_CPU Energy held at 35656018 ... but the probe's sampler read 74528887". The second makes the wall clock the probe's thread clock. It agrees with the thread clock on a quiet host, and fails 3/3 under 36 spinners. - 24 no-feature tests pass. cargo check --tests --examples passes for aarch64-apple-darwin, x86_64-apple-darwin, aarch64-unknown-linux-gnu and x86_64-unknown-linux-gnu, with and without ioreport. clippy --no-deps shows only the warnings that were already there. Not done here - A path that re-implements probe(), by calling probe_with or probe_cpu_rail directly, is not counted. - No CI job runs these ioreport tests; .github/workflows is outside this branch. Co-Authored-By: Claude Opus 5.5 <[email protected]>The tier_store tests sometimes failed with "tier_store.rs:738:50: called `Result::unwrap()` on an `Err` value: Io(Os { code: 2, kind: NotFound, ... })". write_synth_gguf named its file jouleclaw-tier-store-<pid>-<nanos>.gguf. On macOS SystemTime ticks in microseconds, and the harness starts these tests together on parallel threads, so two tests could pick the same name. They then shared one file, and the first to finish deleted it before the other had read it. Each such helper now takes a number from its own process-wide AtomicU64 instead of the time, giving <prefix>-<pid>-<seq>. The crate's mellum, falcon_h1 and hybrid_lm tests already name their files this way. Four more helpers used the same timestamp naming with no per-test tag, and several tests call each of them at once, so they get the same fix: - lib.rs prefetch_policy_tests::write_synth_gguf (8 tests) - gpu_buffer.rs aligned_temp_mmap (5 tests; it truncates and maps the file) - mmap_advise.rs tempfile_compat (2 tests) - moe.rs build_synthetic_store_with_experts (2 tests) None of the four failed in the runs below, but they had the same race. Only #[cfg(test)] code changes. Evidence, on this M5 Max: - Before, on the parent tree (3b260e0a): the lib test binary built with --features fabric-meter, run 30 times, passed 27 and failed 3. All three failures were at tier_store.rs:738:50 with NotFound, in pin_hot_changes_tier_to_hot, demote_to_cold_returns_to_cold and out_of_bounds_errors. - After, the same build and the same 30 runs: 30/30, 299 passed each time. - `cargo test -p jouleclaw-loader-gguf --lib --features fabric-meter`, 12 consecutive runs: 12/12, "test result: ok. 299 passed; 0 failed; 2 ignored". - With --features "rayon fabric-meter": 3/3, 302 passed. - clippy --no-deps --tests --features fabric-meter: no warning on or next to any changed line. It exits 101 on three error-level lints that were already there, in files this commit does not touch: ssm.rs:406 (approx_constant) and gemma4.rs:562 and :582 ("this operation will always return zero"). Co-Authored-By: Claude Opus 5.5 <[email protected]>Round 5 left two dangerous survivors. Tests only ever saw CpuRailFrozen, the one status either Mac produces, so these two passed 57/57 on both Macs: - OWN1: a counter that declined only a frozen rail. - OWN4: a host cache that took "no subscription" for a live rail. It also left four fail-safe survivors: - a constructor that re-implemented the probe, which went uncounted; - a zero or stale transform between the probe's record and the rule; - a tick that always ran to its wall cap; - a latent flake on a host whose rail is live. In the sibling crates, a wall-time median pin also failed on correct code under concurrent all-channel IOReport sampling. This commit closes all of these. The production chain - probe_host_status() is the one production status. It bumps REAL_RUNS, then returns status_of(probe_with(open_probe_sampler, probe_tick, <monotonic clock>, CPU_RAIL_LIVENESS_BUDGET)). It has no branch of its own. - probe_with opens a sampler and watches it. If nothing opened, or no interval was observed, it returns None, which status_of maps to Unavailable. - crate::ioreport_status() now holds the process-wide OnceLock. It is the only production caller of probe_host_status. - AppleIoReportCounter::new() is new_given(ioreport_status()). new_given declines, before opening anything, unless admits(status) says yes. admits is a pure, exhaustive match that is true for Live alone. - GROUP_OPENS counts every call of the real group opener. Only the probe opens the group. So a constructor that re-implements the probe around the cache shows up there, even though it bypasses REAL_RUNS. Both counters are read through #[doc(hidden)] ioreport_probe_runs() and ioreport_group_opens(). Every status is tested - admits: Live admits; Unavailable, CpuRailAbsent and CpuRailFrozen do not. - new_given with Unavailable, CpuRailAbsent or CpuRailFrozen: - opens no subscription of any kind and samples nothing; - is unavailable and names the status; - read() returns NoCounter. - new_given(Live) opens the all-channel subscription once and probes nothing. - OWN1 now fails: "a counter given Unavailable opened a subscription or sampled: Record { group_opens: 0, all_channel_opens: 1, ... }". - The once-per-process pin now lives in its own integration test, tests/ioreport_probe_once_per_process.rs, so it runs in a process of its own. A barrier releases 9 threads: - a third ask for the status first; - a third build a counter first; - a third call detect_meter first. REAL_RUNS and GROUP_OPENS must both end at 1. Without the ioreport backend the test asserts that nothing probed. - In the lib test process the uncached status now runs on purpose, so the unit test there only checks that every thread sees the one cached status. Fault injection inside the production code - A test-only, thread-local seam sits in four places: - inside the real group opener; - inside the all-channel opener; - inside the FFI leaf of the real sampler, sample_raw, now split from the pure energy_model_channels filter; - at the top of RailWatch::observe. - A test can script whether the group opens and what each sample holds. Everything between the leaf and the rule is production code, including the real 25 ms thread-CPU tick and the real clock. - The real, uncached probe_host_status runs under these scripts: - The group does not open: Unavailable, one open call, zero sample calls. OWN4 now fails: "the production status with no Energy Model subscription". - The M5 Max's own single refresh: CpuRailFrozen. - A rail advancing twice: Live, after exactly 3 sample calls. - A constant rail with one failed sample: CpuRailFrozen. The failed sample is kept from the rule, and the verdict comes only after the whole budget. - Two dies stepping in different intervals: CpuRailFrozen. The script also has same-named sub-rails and an out-of-group channel named "CPU Energy" that moves in every sample. Each die reaches the rule at its own value. - No CPU rail: CpuRailAbsent. - Every scripted run also checks two things: - The rule was handed exactly the Energy Model channels of each sample that came back. - Each tick between two sample calls burned at least 25 ms and under 50 ms of this thread's CPU on the tests' clock, unless it hit the 500 ms wall cap. What the rule is handed - The record is now taken at the rule's input (observe), not beside the sampler. A zero or stale transform anywhere between the FFI leaf and the rule now shows up in it. - The freshness test runs the real uncached composition while an oracle reads the group on its own subscription. It checks: - the rule's input equals what the sampler read; - its CPU and GPU rails are the oracle's; - each bracketed sample equals the oracle where the rail held, and lies between the oracle's reads where it moved; - a rail the oracle saw change inside the window also changes in what the rule was handed. - On a host whose rail is live, the probe stops after two intervals (~60 ms), and one such window often holds too few oracle reads. So the test runs the composition up to 8 times, until one run has evidence and a moving rail. - No evidence in 8 runs fails. - Evidence but no moving rail passes, and the test prints that such runs cannot tell a fresh sampler from a stale one. Cost and clocks, with no wall-clock bound - The group-only pin is structural. Every channel of every sample the real probe took is in the Energy Model group. Each sample holds exactly as many channels as the oracle's own group subscription: 364 on the M5 Max, 645 on the M3 Ultra. - The cost pin is the median THREAD CPU per sample call, under 50 ms on the tests' clock. The wall-time median is gone. - No hardware test asserts a wall-clock bound or ratio. The one wall-clock check left is a lower bound in a scripted test: a Frozen verdict comes only after the whole budget. - The tests read their own CLOCK_THREAD_CPUTIME_ID, the literal 16 from the SDK's <_time.h>. Against it, the production clock: - does not advance over a 200 ms sleep (10.5 µs); - does not advance while a helper thread burns 200 ms (57 µs); - advances by what the tests' clock saw over this thread's own burn (100.001 ms for 100.0008 ms). - Each production tick burns at least 25 ms and under 50 ms of this thread's CPU (unless it hits its 500 ms cap), and no more CPU than wall time. - The probe's own record of its ticks and sample costs is gone: ProbeRun, Tick, DELIBERATE_PROBE, NO_ENERGY_MODEL_GROUP and PROBE_RUNS. The tests now measure the ticks themselves, between sample calls. counts_once - Ported from eoc-meter7aa42418. Beside the CPU aggregates, counts_once now also drops: - per-cluster accumulators (PACC*, EACC*, MACC*); - CPU power managers (*CPM*), with their _SRAM rails. It matches on the die-stripped name. The test pins the real channel names of both chips. Evidence (this commit's tree; M5 Max, macOS 27.0.1, hw.ncpu=18) - Tests with --features ioreport: - lib: "test result: ok. 66 passed; 0 failed" (57 before); - ioreport_probe_once_per_process: 1 passed, "real probe runs=1, Energy Model group opens=1"; - vendor_energy_conformance: 1 passed. - Tests without the feature: lib 24, integration 1, conformance 1. - Every probe status line said "ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false". - Load runs, on the lib test binary built from this tree (sha256 df3f59b0…): - 5 runs while a helper process sampled IOReport over all ~11,900 channels in a tight loop (~185 ms a sample): 5/5, each "66 passed". Median sample-call thread CPU 6.95-8.65 ms, median tick 25.071-25.079 ms. - 5 runs under 18 busy-loop spinners (1 x ncpu, load1 14.2 -> 27.3): 5/5. Median sample-call thread CPU 6.59-6.82 ms. - Every helper was killed; kill -0 and pgrep found 0 left. - A light group-only watcher of mine was also running during these runs. - M3 Ultra (10.10.10.2, 27.0.1), the same binaries, run with --nocapture and then deleted: - lib "test result: ok. 66 passed; 0 failed", and the once test "real probe runs=1, Energy Model group opens=1". - 645 channels per sample, equal to the group; median 7.10 ms; DIE_0 and DIE_1 keyed apart. - GPU Energy changed 6 times inside the window and 7 times in what the rule was handed: fresh. - Simulated live host (a temporary edit makes is_live true after two intervals; restored with git checkout), freshness and group-only tests only: - 10/10 unloaded, 10/10 under the all-channel helper. - Under the helper, the median thread CPU over the 3 sample calls was 6.1-8.3 ms; round 5's wall-time median was 100-202 ms there. - Every ~57 ms window held 3-4 oracle reads. - Where no rail moved in any of the 8 windows, the test said it cannot tell fresh from stale: 4 of 10 unloaded, 6 of 10 under the helper. In this simulation the CPU rail is really frozen. On a genuinely live host the rail itself moves in nearly every interval. - Mutations: 22, each an exact-string edit of this tree, run with cargo test --no-fail-fast (lib and integration), then restored with git checkout (clean after each). All 22 failed: - new_given declines only CpuRailFrozen: "a counter given Unavailable opened a subscription or sampled". - no subscription -> Live inside probe_host_status: "the production status with no Energy Model subscription". - a single advance -> Live inside the production closure: "one batch refresh counted as a live rail" and "two dies' separate steps ... made a live rail". - the rule compares with the first sample: 11 tests fail. - a decrease counts as an advance: "a fall counted as an advance". - a no-op tick: "production tick 0 burned 416ns ...". - a tick that always runs to its wall cap: "production tick 0 burned 455.483167ms ... a tick stops once it has burned 25ms". - a per-thread cache: "the real probe ran 10 time(s) and asked for the Energy Model group 10 time(s)". - a constructor that probes uncached: "ran 19 time(s) and asked ... 19". - a constructor that re-implements the probe through probe_with: "ran 1 time(s) and asked for the Energy Model group 19 time(s)". - a zero or stale transform in the production sampler closure: "the rule was not handed exactly the Energy Model channels of each sample". - a zero transform inside watch() before observe: 7 tests fail. - zeroing inside the rule after the record: 6 tests fail. - the sampler reads constant 0: "("CPU Energy", 0) held at 2643843378 across the oracle's reads either side of it, but the rule was handed 0". - the thread clock is CLOCK_MONOTONIC: "this thread slept 200ms, and the production thread clock advanced 210.015ms". - admits accepts CpuRailAbsent: "CpuRailAbsent admitted the counter". - counts_once re-admits PACC_*: "PACC_0, a summand of CPU Energy on an M5 Max, is summed beside it". - extras: - CLOCK_PROCESS_CPUTIME_ID: "advanced 1.767591s". - the probe on the all-channel subscription: 9 lib tests and the integration test fail. - each CPU rail handed to the rule as the sum of all CPU rails: fails on this one-rail M5 Max through the two-die script. - a tick of 30%: "burned 7.50375ms". - Consumers, all passing: - jouledesk-github: 10/10, 12 passed each. - jouleclaw-loader-gguf with fabric-meter: 299 passed, 2 ignored. - jouleclaw-harness-econ: 14. - mesh-meter with hw-meter-apple: 10. - l3-jouleclaw: 2 + 4, with 2 + 2 ignored. - energy_probe prints "ioreport : declined (CpuRailFrozen)". - ~/.jouleclaw/fabric-calibration.json is unchanged (2026-06-11 11:02:14). - cargo check --all-targets, with and without ioreport, on x86_64/aarch64-unknown-linux-gnu and x86_64/aarch64-apple-darwin: 0 warnings. - clippy --no-deps --all-targets --features ioreport shows only warnings that were already there: - the duplicated link attribute, now apple_ioreport.rs:112; - the b"IOReportChannels\0" literal in sum_energy_uj, now :823; - ledger.rs:631-634. - rustdoc --document-private-items shows only its three earlier warnings. Found, not fixed here: one batch refresh is two advances, so the rule can say Live on this frozen host - During the first mutation pass, the per-thread-cache mutant ran about a dozen real probes in one lib test process. Two of them, on two test threads, said Live, and the oracle tests failed: - "probe said Live (probed=CpuRailFrozen, cached=Live), the oracle did not see the CPU rail keep moving"; - "left: [CpuRailFrozen, ...] right: [Live, ...]". - CPU Energy moved in one batch refresh, 2,588,973,125 -> 2,643,843,378, between 17:54 and 17:59:38. That mutant ran from 17:58:24 to 17:58:34. - On this chip CPU Energy is the sum MCPU0 + MCPU1 + PCPU: 311,045,470 + 313,039,186 + 2,019,758,721 = 2,643,843,377, against 2,643,843,378. - A watcher sampled the Energy Model group every ~16 ms: 20 minutes with no refresh, then the next one caught one. The refresh landed as two steps of every cluster and of CPU Energy, 64 ms apart: t=1076.463s CPU Energy 2643843378 -> 2705607863 (+61764485) sample#56065 t=1076.528s CPU Energy 2705607863 -> 2705611298 (+3435) sample#56068 (MCPU0 +9470641 then +788, MCPU1 +9747542 then +844, PCPU +42546301 then +1802.) - The probe's intervals are ~31 ms: a 25 ms tick plus a ~6 ms sample. A probe whose window holds such a refresh sees the same rail advance in two distinct intervals, which is Live under the current rule. - A probe lands on a refresh rarely: its window is 1 s, and the two refreshes seen here came about 30-34 minutes apart. When it does, production caches Live for the whole process. The counter would then sum batch-refreshed CPU rails, plus the GPU, tagged ModelBased: the over-claim this branch exists to stop. - The same mutant on the final tree left the lib tests at 66/66, with no refresh during that run. - The rule itself (the same rail advancing in two distinct intervals) is the cross-crate rule every sibling crate shares, so it is left unchanged here. One direction would demand what a live rail shows and a refresh cannot: a rail that advances in most intervals over a minimum count, for example at least 5 intervals with 80% of them advancing. The M5 Max GPU rail changed in 166 of 176 samples ~28 ms apart. A refresh gives at most 2 or 3 advancing intervals in a row. Not done here - Changing the liveness rule, per the finding above. It needs one decision across jouleclaw-energy, eoc-meter, proof-cascade, sandbox-meter and jcp-energy. - The counter's own read() still samples every channel. It is declined on both Macs. - No CI job runs these ioreport tests: .github/workflows is off-limits to this branch. Co-Authored-By: Claude Opus 5.5 <[email protected]>The CPU-rail liveness rule said Live once one CPU rail had advanced in two separate sample intervals. A macOS 27 batch refresh can meet that on its own. On this M5 Max (macOS 27.0.1, 2026-09-30) a watcher sampling the Energy Model group alone every ~16 ms saw one refresh land in two steps: CPU Energy 2643843378 -> 2705607863 (+61764485) CPU Energy 2705607863 -> 2705611298 (+3435) 64 ms later with MCPU0, MCPU1 and PCPU each stepping twice alongside. The probe's intervals are ~31 ms (a 25 ms CPU tick and a ~6 ms group sample), so a probe window that holds such a refresh sees two advancing intervals and says Live, and that answer is cached for the process. It was seen to happen: under the per-thread-cache mutant, two real probes said Live while the oracle saw no sustained movement, at the moment of a refresh. The rule is now the family rule for the five crates that probe this rail (this commit changes this crate only): - Live iff one CPU rail (same key: name + ordinal) advanced in at least MIN_LIVE_ADVANCES = 5 distinct intervals, those are at least four in five of all the intervals observed so far (advancing * 5 >= intervals * 4), and the samples so far span at least MIN_LIVE_WINDOW = 300 ms of wall-clock time from the first. It is evaluated after every interval, and is Live as soon as all three hold. - The probe keeps sampling until Live, or until the 1 s budget has elapsed and at least 5 intervals were observed; hard cap 5 budgets. - The rule is handed each sample's wall-clock time: `observe(sample, at)`, where the loop passes its clock's reading as each sample comes back, and the tests script it. To be Live, a refresh would have to step in eight of the probe's first ten ~31 ms intervals, which is what a live rail does and a refresh does not. A live rail (the M5 Max GPU rail changed in 13,125 of 13,614 samples 100 ms apart, 96%) is Live after ten unloaded intervals (310 ms), or five 100 ms intervals on a loaded host (500 ms). Everything else is unchanged: per-rail keys, consecutive samples, a decrease or a first appearance is no advance, one advance per rail per interval, the group-only subscription, the thread-CPU tick, the pure verdict-to-status mapping, the OnceLock cache, REAL_RUNS and GROUP_OPENS, the F9 seam (which also records the time handed with each sample now), new_given/admits, counts_once, the consumers, and the "ioreport liveness: status=" line. The docs that described the two-interval rule (module doc, RailWatch, IoReportStatus, ioreport_status, detect_meter, the Apple estimator) describe this one. The test oracle is held to the same bar. It said CpuRailMoved(true) at two changes, so a refresh landing in its window could vouch for a false Live. Now it says so only if one rail changed at five or more of its reads and at four in five or more of them, over its whole window. It reads on past its 3 s window until it has five reads (capped at four windows), and a CPU rail read fewer times still fails the calling test as no evidence. The freshness test's "cannot tell fresh from stale" disclosure is now bounded: a run whose probe said Live must show the oracle seeing a CPU rail change inside the probe's window (Live means the rule was handed one advancing in five or more intervals over 300 ms or more), so the disclosure can only cover runs that were not Live, and a stale sampler can never make one Live. Tests. Those marked (*) fail under the old two-advance rule. - (*) the_observed_two_step_refresh_is_frozen: the refresh above, with its steps in consecutive intervals and 64 ms apart. - (*) a_refresh_in_three_to_five_quick_steps_is_frozen: 3, 4 and 5 steps in the first 150 ms, one per interval. - (*) an_m3_ultra_two_die_two_step_refresh_is_frozen: DIE_0 and DIE_1 each stepping twice within 64 ms. A modelled shape: the M3 Ultra has been seen to refresh in one step per die. - (*) the_production_status_of_a_two_step_refresh_is_frozen: the same refresh through the real uncached host status function (F9 seam). - a_rail_advancing_every_interval_is_live_not_before_five_advances_and_300_ms: Live at interval 10 unloaded (310 ms) and 5 loaded (500 ms); not at 9 (279 ms) or at 3 (300 ms). - a_rail_missing_one_interval_in_five_is_live (Live at exactly 8 of 10) and a_rail_missing_two_intervals_in_five_is_not_yet_live. - the_window_runs_from_the_first_sample_on_the_clock_handed_in, and two_rails_advancing_in_turn_are_not_one_live_rail. - The watch-loop tests at 31 and 100 ms intervals and past the budget: the probe stops at five intervals, keeps going with fewer, stops at the cap. - the_production_status_of_a_rail_that_keeps_advancing_is_live replaces "two advances -> Live after exactly 3 sample calls": Live after 6 to 13 sample calls, each sample's time handed to the rule inside the tests' own clock bracket, at least 300 ms watched, and not Live one sample earlier. - the_oracle_says_moved_only_for_a_rail_that_kept_changing, and the_oracle_reads_on_past_its_window_until_a_live_rail_could_meet_its_bar. The lib tests go from 66 to 76. Mutations, each a temporary edit restored with git checkout (the tree clean after each), lib tests with --features ioreport: - the old rule (two advances, nothing else): 13 fail, all four (*) among them - MIN_LIVE_ADVANCES = 2: 6 fail - the four-in-five check removed: 3 fail - the 300 ms window check removed: 6 fail - the oracle back to ">= 2 changes": 2 fail - also caught: the window measured from the clock's origin (3 fail), the loop handing the rule a zero time (3), ">" for ">=" in the share (2). Runs. jouleclaw-energy lib and integration tests with and without --features ioreport (76 + 1 + 1, and 24 + 1 + 1). Three times beside an all-channel IOReport sampler looping in another process, all green: ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false oracle_reads=7 ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false oracle_reads=5 ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false oracle_reads=5 The oracle extended itself to five reads in up to 5.5 s where its 3 s window gave fewer. On the M3 Ultra (macOS 27.0.1), 76 passed: ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false oracle_reads=16 The scripted sustained rail went Live after 13 sample calls there and in every M5 Max run (rule window 302-322 ms). The consumers pass: jouledesk-github (12), jouleclaw-loader-gguf --features fabric-meter (367), jouleclaw-harness-econ (14), mesh-meter --features hw-meter-apple (10), l3-jouleclaw-binding (6). clippy --no-deps --features ioreport gives the same six warnings as before, none new; rustdoc adds none. Not done here: - The freshness test's Live assertion never runs on either Mac here, since both are CpuRailFrozen; it guards a live host. - The budget rule stops at five intervals once 1 s has passed, even if the window is still under 300 ms. That only happens if the first sample took more than ~700 ms, and it declines a live rail, which is the safe side. Co-Authored-By: Claude Opus 5.5 <[email protected]>WIP: jouleclaw-energy: IOReport only when its CPU rail keeps moving; consumers never zero-fill or relabelto jouleclaw-energy: IOReport only when its CPU rail keeps moving; consumers never zero-fill or relabel`PassportBuilder::build` accepted any pairing of `measurement_method` with `compliance.eu_ai_act.inference_energy_measured`. That includes `IoReport`, which `MeasurementMethod::provenance` itself grades `Estimator`, paired with `inference_energy_measured: true`. A sealed passport could then disclose estimator energy as measured inference energy under the EU AI Act field. It also accepted `Estimated` with `true`. The rationale beside `IoReport` was "estimator until the IOReport FFI ships". The FFI shipped long ago. The real reason, after the previous commits: a vendor energy counter is a real reading, and what decides its grade is provenance. Was the counter live for the domain the figure claims? A bare `IoReport` records no such observation, and where the CPU rail is frozen (macOS 27.0.1, for an unprivileged reader) the host's figure is the estimator's. So `IoReport` stays `Estimator`, and the stale wording is replaced in the variant doc, the crate doc and the test. The fix. - New `MeasurementMethod::IoReportLive`: IOReport read while its CPU energy rail was observed live (`IoReportStatus::Live`). It is graded `HwShunt`, like RAPL and NVML. Display: "IOReport (live)". It is additive, so existing serialized passports still deserialize. - `is_live_hardware_reading()` is true iff the method's grade is `HwShunt`. - `build()` returns `PassportError::MeasuredWithoutHardwareReading { method, provenance }` when `inference_energy_measured` is true and the method is not a live hardware reading. That means a bare `IoReport` or `Estimated`. Unclaimed, every method still builds. Tests: `a_measured_claim_needs_a_live_hardware_reading`. `IoReport` and `Estimated` with a measured claim are refused with their method and grade, and build without it. `IoReportLive`, `Rapl`, `Nvml` and `NvmlRapl` build with the claim. The message names the method, the grade and the rule. The mapping test covers `IoReportLive` → `HwShunt`, and the display test covers its label. jouleclaw-compliance: 53 pass. Falsified (restored with `git checkout`, tree clean): a gate that refuses only `Estimated`, and so accepts `IoReport` with a measured claim, fails the test with "IOReport paired with a measured claim was not refused". 52 / 1. Co-Authored-By: Claude Opus 5.5 <[email protected]>