From 04086d2e386ee5972db2c1584980767199155e7f Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Mon, 19 Oct 2026 20:26:38 -0400 Subject: [PATCH] jouleclaw: bound a GPU energy counter from below, not only above MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `GpuPlausibility` judged an NVML cumulative-energy counter against the board's own telemetry in one direction only. Its three verdicts were `Plausible`, `CounterOverRate` ("ran faster than the board can draw") and `BoardTelemetryUnusable`. Nothing caught a counter reading **low**. That is the wrong direction to guard. By this family's own doctrine (jcp/SPEC §5): "Under-reporting is the unsafe direction at a ceiling, because it admits work the budget cannot cover." A counter running fast fails safe — it spends a budget sooner. A counter reading low is graded `HwShunt` and then spends an energy ceiling it is not measuring. jouleclaw/SPEC §1.3 already refused a counter "republished in lumps", but nothing implemented it: such a counter *advances*, so the sustained-advance liveness probe calls it live while it accrues a fraction of the energy. `CounterUnderRate` closes it, bounded by the **same** board telemetry the ceiling bound already uses, so no threshold is invented: floor = (mean_usable_draw - GPU_DRAW_SLACK_MW) / GPU_UNDER_FACTOR The slack comes off the draw *before* the division, so the floor collapses to zero at or below 5 W of mean draw: on an idle board a correct counter accrues almost nothing, and refuting it would be noise rather than a finding. The factor is deliberately looser than `GPU_DRAW_FACTOR` because the draw readings are one-second averages, so a bursty load idle between samples really does integrate to less than its sampled mean. The verdict carries the yardstick it was judged against, so the judgement can be re-made against a different factor. It ranks **above** a proven overrun: both refute, but only this one admits work a budget cannot cover, so where one GPU of a probe over-reports and another under-reports, the under-report is what an operator must act on. Among two under-reporting GPUs the ordering is by shortfall, not by rate. Two things found while doing it, beyond the reported gap: - **`_ => CounterLiveness::Live` in the verdict mapping** meant a newly added verdict would silently become *live* — and "unrecognised verdict is admissible" is the unsafe default. That match is now exhaustive, so a future arm must be graded deliberately or the crate does not compile. - **The wiring was untested.** The existing test covered `plausible_gpu_counter` and the over-rate path through `GpuWatch::liveness`, but a verdict that never reaches `CounterLiveness` changes nothing. Added a test driving an under-reporting GPU through the probe, whose discriminating half is the *same slow counter beside an idle board* — which must stay live, because nothing about the counter changed, only the yardstick. `mode_for` needed no change: `Implausible` already falls back to the vendor power integral and grades `ModelBased`, so the fix completes itself once the verdict exists and maps. SPEC §1.3 now states the bound is two-sided, that it refutes and never verifies, that an indeterminate is not hardware-graded, and that it must not fire near idle. Doc comments that described the bound as one-directional are corrected. Figures throughout are the jetson-hub RTX 4050's own, as the existing test's already are. Co-Authored-By: Claude Opus 5 --- jouleclaw/SPEC.md | 21 ++ .../crates/jouleclaw-energy/src/liveness.rs | 211 +++++++++++++++++- .../crates/jouleclaw-energy/src/nvml.rs | 16 +- 3 files changed, 238 insertions(+), 10 deletions(-) diff --git a/jouleclaw/SPEC.md b/jouleclaw/SPEC.md index 47db2db4d..1d8749023 100644 --- a/jouleclaw/SPEC.md +++ b/jouleclaw/SPEC.md @@ -94,6 +94,27 @@ unmetered, and **SHOULD** record why. A figure **MUST NOT** claim coverage of a domain its counter does not read; where the format has a coverage field, such a figure is Partial. +**The bound is two-sided (MUST).** Where an implementation judges a vendor +energy counter against an independent yardstick — the board's own power +telemetry, a datasheet envelope — it **MUST** apply the bound in **both** +directions. A counter that accrues far *less* than the yardstick says the +device drew is as refuted as one that accrues far more, and it is the more +dangerous of the two: **under-reporting is the unsafe direction at an energy +ceiling, because it admits work the budget cannot cover**, while a counter +running fast fails safe by spending a budget sooner. A counter republished in +lumps, sampled across its republish period, reads low rather than frozen — it +advances, so a liveness probe alone calls it live. + +Such a judgement **refutes, never verifies**. The yardstick is itself usually +`ModelBased` (a vendor power estimate), so a counter inside both bounds is *not +refuted*, which is not a finding that it is correct. Where the yardstick is +absent or itself unusable the result is an **indeterminate**, and an +indeterminate **MUST NOT** be graded as a hardware reading either: erring +toward declining is the safe direction. A bound **MUST NOT** fire where the +device is near idle and the yardstick leaves no room for a finding — at idle +draw a correctly reporting counter accrues almost nothing, and refuting it +would be noise, not a defect. + Apple IOReport's GPU rail is not a partial reading of CPU work. It is machine-wide — it meters every client of the GPU on the host, display compositing among them, and moves while nothing of the metered work diff --git a/jouleclaw/jouleclaw-rs/crates/jouleclaw-energy/src/liveness.rs b/jouleclaw/jouleclaw-rs/crates/jouleclaw-energy/src/liveness.rs index 887ff1dd0..6ee4c1324 100644 --- a/jouleclaw/jouleclaw-rs/crates/jouleclaw-energy/src/liveness.rs +++ b/jouleclaw/jouleclaw-rs/crates/jouleclaw-energy/src/liveness.rs @@ -337,6 +337,24 @@ pub(crate) const GPU_DRAW_FACTOR: u128 = 2; #[cfg_attr(not(any(test, feature = "nvml")), allow(dead_code))] // used by `nvml` pub(crate) const GPU_DRAW_SLACK_MW: u128 = 5_000; +/// How far **below** its board's mean draw a counter may accrue before it is +/// refuted: a shortfall of more than this factor, after [`GPU_DRAW_SLACK_MW`] +/// is taken off the draw, is no count of the board's energy. +/// +/// The bound has to exist in this direction at all because **under-reporting +/// is the unsafe direction at an energy ceiling** — it admits work the budget +/// cannot cover — while the ceiling bounds only catch a counter running *fast*, +/// which is the direction that fails safe. SPEC §1.3 already refuses a counter +/// "republished in lumps", and a counter sampled across its republish period +/// reads low rather than frozen, so the liveness probe alone does not see it. +/// +/// It is deliberately looser than [`GPU_DRAW_FACTOR`]. The draw readings are +/// one-second averages, so a bursty load that is idle between samples really +/// does integrate to less than its sampled mean, and the slack subtraction +/// means the floor collapses to zero at or below [`GPU_DRAW_SLACK_MW`]: at +/// idle draw there is no finding to make, only noise to avoid. +pub(crate) const GPU_UNDER_FACTOR: u128 = 4; + /// Whether one GPU's energy counter, which counted `energy_mj` over `window`, /// plausibly counted its board's energy, beside the board's power draw /// `draws_mw` read with the samples and its maximum power limit `ceiling_mw`. @@ -387,6 +405,24 @@ pub(crate) enum GpuPlausibility { /// Which bound that was. bound: GpuBound, }, + /// The counter accrued far less energy than the board's own telemetry says + /// it drew. + /// + /// This is the finding the ceiling bounds cannot make, and the dangerous + /// one: a counter reading low is graded `HwShunt` and then spends a budget + /// it is not measuring. A counter republished in lumps and sampled across + /// the republish period looks exactly like this — it advances, so the + /// liveness probe calls it live, while accruing a fraction of the energy. + CounterUnderRate { + /// The rate the counter implies over the window, in milliwatts. + rate_mw: u64, + /// The floor it fell below, in milliwatts. + floor_mw: u64, + /// The board's mean usable draw the floor was taken from, in + /// milliwatts — the yardstick, so the judgement can be re-made against + /// a different factor. + mean_draw_mw: u64, + }, /// The board's own telemetry read past the board's own power limit, so /// there is no yardstick left to judge the counter against. An /// indeterminate, not a finding about the counter — see @@ -421,6 +457,11 @@ impl GpuPlausibility { GpuPlausibility::Plausible { .. } => 0, GpuPlausibility::BoardTelemetryUnusable { .. } => 1, GpuPlausibility::CounterOverRate { .. } => 2, + // Above a proven overrun: both are refutations, but only this one + // is the direction that admits work a budget cannot cover, so where + // one GPU of a probe over-reports and another under-reports, the + // under-report is the verdict an operator has to act on. + GpuPlausibility::CounterUnderRate { .. } => 3, } } @@ -429,6 +470,12 @@ impl GpuPlausibility { match self { GpuPlausibility::Plausible { rate_mw } => *rate_mw, GpuPlausibility::CounterOverRate { rate_mw, .. } => *rate_mw, + // The shortfall, so that among two under-reporting GPUs the worse + // one speaks: a lower rate is a bigger shortfall, so it is ordered + // by how far the floor was missed rather than by the rate itself. + GpuPlausibility::CounterUnderRate { rate_mw, floor_mw, .. } => { + floor_mw.saturating_sub(*rate_mw) + } GpuPlausibility::BoardTelemetryUnusable { draw_mw, .. } => *draw_mw, } } @@ -493,6 +540,17 @@ pub(crate) fn plausible_gpu_counter( bound: GpuBound::MeanDraw, }; } + // And the same yardstick from below. The slack comes off the draw + // before the division, so the floor is zero at or under + // `GPU_DRAW_SLACK_MW` of mean draw: an idle board makes no finding. + let floor = mean.saturating_sub(GPU_DRAW_SLACK_MW) / GPU_UNDER_FACTOR; + if floor > 0 && rate_mw < floor { + return GpuPlausibility::CounterUnderRate { + rate_mw: clamp(rate_mw), + floor_mw: clamp(floor), + mean_draw_mw: clamp(mean), + }; + } } GpuPlausibility::Plausible { rate_mw: clamp(rate_mw) } } @@ -529,8 +587,10 @@ impl GpuWatch { /// except that a live counter is re-judged on whether its GPUs counted /// their boards' energy over the probe, beside each GPU's maximum power /// limit `ceilings_mw`: a counter proven to run faster than its board can - /// draw is [`CounterLiveness::Implausible`], and a board whose own - /// telemetry is unusable leaves it [`CounterLiveness::Unjudgeable`]. + /// draw — **or far slower than it** — is [`CounterLiveness::Implausible`], + /// and a board whose own telemetry is unusable leaves it + /// [`CounterLiveness::Unjudgeable`]. Both directions refute, because a + /// counter reading low spends an energy ceiling it is not measuring. pub(crate) fn liveness(&self, verdict: CounterLiveness, ceilings_mw: &[Option]) -> CounterLiveness { let (Some((first, t0)), Some((last, t1))) = (&self.first, &self.last) else { return verdict; @@ -557,10 +617,16 @@ impl GpuWatch { ) }) .max_by_key(|v| (v.rank(), v.magnitude())); + // Exhaustive by intent. This was `_ => Live`, which would have made a + // newly added verdict silently admissible — and "unrecognised verdict + // becomes live" is the unsafe direction. A new arm must now be given a + // liveness deliberately, or the crate does not compile. match worst { - Some(GpuPlausibility::CounterOverRate { .. }) => CounterLiveness::Implausible, + Some( + GpuPlausibility::CounterOverRate { .. } | GpuPlausibility::CounterUnderRate { .. }, + ) => CounterLiveness::Implausible, Some(GpuPlausibility::BoardTelemetryUnusable { .. }) => CounterLiveness::Unjudgeable, - _ => CounterLiveness::Live, + Some(GpuPlausibility::Plausible { .. }) | None => CounterLiveness::Live, } } } @@ -578,6 +644,99 @@ pub(crate) mod tests { const TEST_MIN_ADVANCES: u32 = 5; const TEST_MIN_WINDOW: Duration = Duration::from_millis(300); + /// **The bound that was missing: a counter reading LOW.** + /// + /// The ceiling bounds catch a counter running faster than its board can + /// draw, which is the direction that fails safe — an over-reported figure + /// spends a budget sooner. A counter reading *low* is the dangerous one: it + /// is graded `HwShunt` and then spends a ceiling it is not measuring, which + /// SPEC §1.3 refuses ("republished in lumps"). Such a counter still + /// *advances*, so the liveness probe calls it live. + /// + /// Figures are the jetson-hub RTX 4050's own: a board drawing 14.8 W while + /// the counter accrues as if it were drawing a fraction of that. + #[test] + fn a_gpu_counter_must_count_no_slower_than_its_board_draws() { + let w = Duration::from_millis(500); + // Over a 500 ms window a counter's rate in mW is twice its mJ. + let draws = &[14_800u32, 14_800]; + // floor = (14_800 - 5_000) / 4 = 2_450 mW. + + // A counter accruing 0.5 W against a 14.8 W board: refuted, with the + // yardstick carried so the judgement can be re-made. + assert_eq!( + plausible_gpu_counter(250, w, draws, Some(65_000)), + GpuPlausibility::CounterUnderRate { + rate_mw: 500, + floor_mw: 2_450, + mean_draw_mw: 14_800 + }, + "0.5 W counted against a 14.8 W draw is no count of the board's energy" + ); + + // A counter that merely lags its board still passes: the draws are + // one-second averages, so a bursty load really does integrate to less + // than its sampled mean. + assert!( + plausible_gpu_counter(1_300, w, draws, Some(65_000)).plausible(), + "2.6 W against a 14.8 W mean is within the factor and must pass" + ); + // Exactly at the floor is not below it. + assert!(plausible_gpu_counter(1_225, w, draws, Some(65_000)).plausible()); + } + + /// An idle board makes no finding. The slack comes off the draw before the + /// division, so the floor collapses to zero at or below it — otherwise a + /// correctly-reporting counter on an idle GPU would be refuted for + /// accruing almost nothing, which is exactly what it should accrue. + #[test] + fn an_idle_board_yields_no_under_rate_finding() { + let w = Duration::from_millis(500); + for mean in [0u32, 1_000, 4_000, 5_000] { + assert!( + plausible_gpu_counter(0, w, &[mean, mean], Some(65_000)).plausible(), + "a counter accruing nothing beside a {mean} mW board must not be refuted" + ); + } + // Just above the slack the floor is barely open, and only a near-dead + // counter falls through it. + let just_above = plausible_gpu_counter(0, w, &[9_000, 9_000], Some(65_000)); + assert_eq!( + just_above, + GpuPlausibility::CounterUnderRate { rate_mw: 0, floor_mw: 1_000, mean_draw_mw: 9_000 } + ); + } + + /// With no usable board telemetry there is no yardstick, so there is no + /// finding in either direction — an indeterminate is not a refutation. + #[test] + fn no_telemetry_means_no_under_rate_finding() { + let w = Duration::from_millis(500); + assert!( + plausible_gpu_counter(0, w, &[], Some(65_000)).plausible(), + "no draw readings: the bound is not applied" + ); + // Every reading past the board's own limit: unjudgeable, not refuted. + assert!(matches!( + plausible_gpu_counter(0, w, &[590_000], Some(65_000)), + GpuPlausibility::BoardTelemetryUnusable { .. } + )); + } + + /// Under-reporting outranks over-reporting, so across a probe's GPUs the + /// verdict an operator must act on is the one that admits unbudgeted work. + #[test] + fn an_under_reporting_gpu_speaks_for_a_probe_before_an_over_reporting_one() { + let w = Duration::from_millis(500); + let under = plausible_gpu_counter(250, w, &[14_800], Some(65_000)); + let over = plausible_gpu_counter(152_250, w, &[5_550], Some(65_000)); + assert!(matches!(under, GpuPlausibility::CounterUnderRate { .. })); + assert!(matches!(over, GpuPlausibility::CounterOverRate { .. })); + let worst = [over, under].into_iter().max_by_key(|v| (v.rank(), v.magnitude())); + assert_eq!(worst, Some(under), "the unsafe direction is the one that speaks"); + assert!(!under.plausible(), "and it is not admissible"); + } + /// The plausibility rule on one GPU counter, with the readings of the RTX /// 4050 Laptop GPU on jetson-hub (driver 595.84, 65 W maximum limit): /// polled at the NVML probe's 100 ms tick its counter ran at 58-305 W while @@ -702,6 +861,50 @@ pub(crate) mod tests { assert_eq!(GpuWatch::default().liveness(Live, &ceilings), Live, "no sample"); } + /// **The wiring, not just the verdict.** A refutation that never reaches + /// [`CounterLiveness`] changes nothing: the counter would still be graded + /// `HwShunt` and still spend a ceiling it is not measuring. So this pins + /// the path from an under-reporting GPU through `GpuWatch::liveness`. + /// + /// GPU 0's counter rises 0.05 J over the 500 ms probe — 100 mW — while its + /// board reports drawing 14.8 W. It advances in every interval, so the + /// sustained-advance rule calls it live; only the lower bound catches it. + #[test] + fn an_under_reporting_gpu_makes_the_probe_implausible() { + let t0 = Instant::now(); + let watch = |draws: [Option; 2]| { + let mut g = GpuWatch::default(); + for k in 0..6u64 { + // GPU 0: +10 mJ per 100 ms interval (100 mW). GPU 1: 1 J (10 W). + let uj = [8_000_000_000 + 10_000 * k, 9_000_000_000 + 1_000_000 * k]; + g.saw(Some(&uj), &draws, t0 + Duration::from_millis(100) * k as u32); + } + g + }; + let ceilings = [Some(65_000), Some(65_000)]; + use CounterLiveness::*; + + assert_eq!( + watch([Some(14_800), Some(10_000)]).liveness(Live, &ceilings), + Implausible, + "GPU 0 counted 0.1 W while its board drew 14.8 W: live, and not measuring it" + ); + + // The discriminating half: the same slow counter beside an *idle* + // board is exactly what a correct counter looks like, and must stay + // live. Nothing about the counter changed — only the yardstick. + assert_eq!( + watch([Some(2_000), Some(10_000)]).liveness(Live, &ceilings), + Live, + "0.1 W counted beside a 2 W board is a counter doing its job" + ); + + // And a verdict that was not `Live` still stands, whatever the draws. + for verdict in [Frozen, Unavailable, Unjudgeable] { + assert_eq!(watch([Some(14_800), Some(10_000)]).liveness(verdict, &ceilings), verdict); + } + } + /// Feed a fresh rule `samples`, the i-th at `i * every`; return it and /// whether it was live after each interval. fn feed(channels: usize, samples: &[Vec], every: Duration) -> (AdvanceWatch, Vec) { diff --git a/jouleclaw/jouleclaw-rs/crates/jouleclaw-energy/src/nvml.rs b/jouleclaw/jouleclaw-rs/crates/jouleclaw-energy/src/nvml.rs index e4cd7c82f..80e12b031 100644 --- a/jouleclaw/jouleclaw-rs/crates/jouleclaw-energy/src/nvml.rs +++ b/jouleclaw/jouleclaw-rs/crates/jouleclaw-energy/src/nvml.rs @@ -12,11 +12,15 @@ //! 4050 Laptop GPU, driver 595.84, republishes the counter every ~100 ms, so a //! live counter has moved in nearly every interval; even at a 2 W idle that is //! ~0.2 J an interval, against a 1 mJ quantum). And every GPU's counter must -//! plausibly count its board's energy over the probe: no faster than 1.5 -//! times the board's maximum power limit, or than twice the board's own power -//! draw plus 5 W ([`crate::CounterLiveness::Implausible`]). That RTX 4050's -//! counter runs at 4 to 30 times its board's draw while it is polled, so it -//! is declined there. Probed once per process and cached. +//! plausibly count its board's energy over the probe, bounded in **both** +//! directions: no faster than 1.5 times the board's maximum power limit, or +//! than twice the board's own power draw plus 5 W, and no slower than a +//! quarter of that draw once 5 W of slack is taken off it +//! ([`crate::CounterLiveness::Implausible`]). The lower bound is the one that +//! matters at a ceiling — a counter reading low is graded as hardware and then +//! spends a budget it is not measuring — while a counter running fast fails +//! safe. That RTX 4050's counter runs at 4 to 30 times its board's draw while +//! it is polled, so it is declined there. Probed once per process and cached. //! //! Where that counter is unsupported (pre-Volta, some consumer parts) or not //! live, the counter integrates `power_usage()` over wall time instead. That @@ -194,7 +198,7 @@ impl NvmlCounter { /// The real liveness probe of every GPU's cumulative-energy counter: the /// sustained-advance rule at [`PROBE_TICK`], with every GPU's power draw read /// at each sample and its maximum power limit once, so that a live counter -/// which ran faster than its board can draw is +/// which ran faster than its board can draw, or far slower than it, is /// [`CounterLiveness::Implausible`] ([`crate::liveness::GpuWatch`]). fn probe_liveness(nvml: &nvml_wrapper::Nvml, device_count: u32) -> CounterLiveness { let ceilings: Vec> = (0..device_count) -- 2.54.0 (Apple Git-157)