jcp/SPEC.md

@@ -404,6 +404,46 @@ its estimator or reports the work unmetered, and SHOULD record why. A figure
 MUST NOT claim coverage of a domain its counter does not read; where the format
 has a coverage field, such a figure is Partial.
 
+**Plausibility (REQUIRED where a bound is readable).** Liveness is necessary and
+not sufficient: a counter can advance continuously and still not be counting
+what it claims. An NVML total-energy counter on an RTX 4050 Laptop GPU (driver
+595.84) advances in every sample interval while running at four to thirty times
+the board's own power draw for as long as it is polled — perfectly live, and no
+count of that board's energy. A figure from it would debit energy the device
+never spent.
+
+So where the platform exposes a bound that is read **independently of the
+counter** — the board's maximum power limit, the board's instantaneous power
+draw — a hardware-graded figure's implied rate MUST be checked against it, and a
+counter whose rate exceeds the bound MUST NOT produce a hardware-graded figure.
+An implementation SHOULD record the rate and the bound it broke, not merely that
+it broke one: a counter a tenth over a bound and one twenty times over are the
+same refusal and not the same finding, and re-grading a past refusal against a
+revised bound is only possible if the rate survived the check.
+
+A bound reading that is itself impossible is not a reading: the same RTX 4050
+reports about 590 W, past its own 65 W limit, for a second after it wakes. Such a
+reading MUST be discarded — and only that reading. One broken sample is not a
+broken bracket, and discarding the whole judgement on it reports an indeterminate
+where the evidence supports a finding: on the reference hardware a single spike
+among otherwise sane 14.8 W samples would otherwise excuse a counter running
+persistently at four to thirty times the board's draw. Discarding them
+individually also keeps the comparison honest, since a spike left in a mean makes
+the bound it derives *more* permissive.
+
+Only where **no usable reading remains** is the counter **unjudgeable** rather
+than implausible: there is then no yardstick, so there is no finding about the
+counter. It MUST NOT produce a hardware-graded figure either, because unjudgeable
+is not admissible and declining is the safe direction, but an implementation MUST
+NOT report it as a counter that overran. The two call for different operator
+responses — replacing a counter, or waiting for a board to settle — and
+conflating them loses that. A bound that is a static limit rather than a reading
+(the board's maximum power limit) is never discarded, so a broken draw sample
+cannot excuse a counter that overran that limit outright.
+
+Where no such bound is readable, this requirement does not apply; liveness
+stands alone.
+
 The record of why is the implementation's to choose; the reference meter
 exposes each counter's liveness status, and which rung it selected with the
 status of every rung above it that it declined. Coverage

jcp/jcp-rs/crates/jcp-energy/src/lib.rs

@@ -646,13 +646,30 @@ pub enum CounterStatus {
     Frozen,
     /// The counter kept advancing, but a GPU counter (NVML's) ran faster than
     /// its board can draw power over the probe: past 1.5 times the board's
-    /// maximum power limit, past twice the board's own power draw plus 5 W,
-    /// or with the board's draw reading itself past that limit. It is not
-    /// counting the board's energy, so a figure from it would debit energy
-    /// the device never spent. Seen on an RTX 4050 Laptop GPU (driver
+    /// maximum power limit, or past twice the board's own power draw plus 5 W.
+    /// It is not counting the board's energy, so a figure from it would debit
+    /// energy the device never spent. Seen on an RTX 4050 Laptop GPU (driver
     /// 595.84), whose counter runs at 4 to 30 times the board's draw while it
     /// is polled. Never selected.
+    ///
+    /// By how much it overran is in [`GpuPlausibility::CounterOverRate`], which
+    /// carries the rate and the bound it broke: a counter 1.1 times over a
+    /// bound and one 20 times over are the same status but not the same
+    /// finding, and tuning the margins later needs the figures.
     Implausible,
+    /// The counter kept advancing and could not be judged, because the board's
+    /// own power telemetry was unusable over the probe: a draw reading past the
+    /// board's own maximum power limit (an RTX 4050 Laptop GPU reports about
+    /// 590 W for a second after it wakes).
+    ///
+    /// Split out from [`CounterStatus::Implausible`], which used to cover this
+    /// and whose doc then asserted "it is not counting the board's energy" —
+    /// a conclusion this case does not support. The yardstick is broken, not
+    /// necessarily the counter. Never selected either way: unjudgeable is not
+    /// admissible, and erring toward declining is the safe direction. The
+    /// distinction is the diagnosis an operator gets, which is the difference
+    /// between replacing a counter and waiting for the board to settle.
+    Unjudgeable,
     /// Every rail of the counter kept advancing while the probe ran: each in
     /// at least five separate sample intervals, making up at least 80% of
     /// those observed, over at least 300 ms. The rung is selected.
@@ -855,23 +872,122 @@ const GPU_DRAW_SLACK_MW: u128 = 5_000;
 /// 595.84) the total-energy counter runs several times faster than the
 /// board draws for as long as it is polled: at this probe's 100 ms wait it
 /// ran at 58-305 W while the board drew 5-14 W against a 65 W limit.
-fn plausible_gpu_counter(energy_mj: u64, window: std::time::Duration, draws_mw: &[u32], ceiling_mw: Option<u32>) -> bool {
+/// Which bound a GPU counter's rate broke.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum GpuBound {
+    /// [`GPU_CEILING_MARGIN`] of the board's maximum power limit.
+    PowerLimit,
+    /// [`GPU_DRAW_FACTOR`] times the board's mean draw plus
+    /// [`GPU_DRAW_SLACK_MW`].
+    MeanDraw,
+}
+
+/// What the plausibility check found, with the figures it found it from.
+///
+/// A `bool` was the whole of this, and it threw away everything an operator or
+/// an auditor needs: which of the bounds broke, by how much, and the rate it
+/// broke them at. A counter 1.1 times over a bound and one 20 times over were
+/// the same answer. The figures also make the judgement reproducible — the
+/// margins are constants in this crate, and re-grading a past finding against
+/// a different margin is only possible if the rate survived the check.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum GpuPlausibility {
+    /// The counter's rate sits inside every bound whose reading was present.
+    Plausible {
+        /// The rate the counter implies over the window, in milliwatts.
+        rate_mw: u64,
+    },
+    /// The counter ran faster than the board can draw.
+    CounterOverRate {
+        /// The rate the counter implies over the window, in milliwatts.
+        rate_mw: u64,
+        /// The bound it broke, in milliwatts.
+        bound_mw: u64,
+        /// Which bound that was.
+        bound: GpuBound,
+    },
+    /// The board's own telemetry read past the board's own power limit, so
+    /// there is no yardstick to judge the counter against. An indeterminate,
+    /// not a finding about the counter — see [`CounterStatus::Unjudgeable`].
+    BoardTelemetryUnusable {
+        /// The offending draw reading, in milliwatts.
+        draw_mw: u64,
+        /// The limit it exceeded (the margin-adjusted cap), in milliwatts.
+        limit_mw: u64,
+    },
+}
+
+impl GpuPlausibility {
+    /// Whether the counter was judged to be counting its board's energy.
+    ///
+    /// `false` for an indeterminate as well as for an overrun: unjudgeable is
+    /// not admissible, and erring toward declining is the safe direction. Use
+    /// the variant itself to tell the two apart.
+    pub fn plausible(&self) -> bool {
+        matches!(self, GpuPlausibility::Plausible { .. })
+    }
+}
+
+fn plausible_gpu_counter(
+    energy_mj: u64,
+    window: std::time::Duration,
+    draws_mw: &[u32],
+    ceiling_mw: Option<u32>,
+) -> GpuPlausibility {
     let ns = window.as_nanos().max(1);
     // mJ per second is mW.
     let rate_mw = u128::from(energy_mj) * 1_000_000_000 / ns;
-    if let Some(limit) = ceiling_mw.filter(|&l| l > 0) {
-        let cap = u128::from(limit) * GPU_CEILING_MARGIN.0 / GPU_CEILING_MARGIN.1;
-        if rate_mw > cap || draws_mw.iter().any(|&d| u128::from(d) > cap) {
-            return false;
+    let clamp = |v: u128| u64::try_from(v).unwrap_or(u64::MAX);
+    let cap = ceiling_mw
+        .filter(|&l| l > 0)
+        .map(|l| u128::from(l) * GPU_CEILING_MARGIN.0 / GPU_CEILING_MARGIN.1);
+
+    // A draw reading past the board's own limit is not a reading: an RTX 4050
+    // Laptop GPU reports about 590 W for a second after it wakes. Such a
+    // reading is dropped, and only that reading — it is one broken sample, not
+    // a broken bracket. Voiding the whole judgement on it was the first shape
+    // of this code and the hardware test refused it: on jetson-hub one spike
+    // among otherwise sane 14.8 W samples turned a counter running
+    // persistently at 4-30x the board's draw into an indeterminate, which is a
+    // worse diagnosis than the overrun it actually is.
+    //
+    // Dropping them also keeps the mean honest. The old code summed every
+    // reading, so a 590 W spike inflated the mean and made the mean-draw bound
+    // *more* permissive — it only ever went unnoticed because any such reading
+    // short-circuited the whole check first.
+    let (usable, unusable): (Vec<u128>, Vec<u128>) = draws_mw
+        .iter()
+        .map(|&d| u128::from(d))
+        .partition(|&d| cap.is_none_or(|c| d <= c));
+
+    if let Some(cap) = cap {
+        if rate_mw > cap {
+            return GpuPlausibility::CounterOverRate {
+                rate_mw: clamp(rate_mw),
+                bound_mw: clamp(cap),
+                bound: GpuBound::PowerLimit,
+            };
+        }
+        // Only when nothing usable is left is there no yardstick at all.
+        if usable.is_empty() && !unusable.is_empty() {
+            return GpuPlausibility::BoardTelemetryUnusable {
+                draw_mw: clamp(unusable[0]),
+                limit_mw: clamp(cap),
+            };
         }
     }
-    if !draws_mw.is_empty() {
-        let mean = draws_mw.iter().map(|&d| u128::from(d)).sum::<u128>() / draws_mw.len() as u128;
-        if rate_mw > GPU_DRAW_FACTOR * mean + GPU_DRAW_SLACK_MW {
-            return false;
+    if !usable.is_empty() {
+        let mean = usable.iter().sum::<u128>() / usable.len() as u128;
+        let bound = GPU_DRAW_FACTOR * mean + GPU_DRAW_SLACK_MW;
+        if rate_mw > bound {
+            return GpuPlausibility::CounterOverRate {
+                rate_mw: clamp(rate_mw),
+                bound_mw: clamp(bound),
+                bound: GpuBound::MeanDraw,
+            };
         }
     }
-    true
+    GpuPlausibility::Plausible { rate_mw: clamp(rate_mw) }
 }
 
 /// The status an NVML probe reports: [`liveness::status_of`] its run, except
@@ -880,15 +996,24 @@ fn plausible_gpu_counter(energy_mj: u64, window: std::time::Duration, draws_mw:
 #[cfg_attr(not(any(test, feature = "nvml")), allow(dead_code))]
 fn nvml_status_of(probed: Option<&liveness::Advances>, gpu: &GpuWatch) -> CounterStatus {
     match liveness::status_of(probed) {
-        CounterStatus::Live => {
-            let plausible = match (gpu.first, gpu.last) {
-                (Some((a, t0)), Some((b, t1))) => {
-                    plausible_gpu_counter(b.saturating_sub(a), t1.saturating_duration_since(t0), &gpu.draws_mw, gpu.ceiling_mw)
+        CounterStatus::Live => match (gpu.first, gpu.last) {
+            (Some((a, t0)), Some((b, t1))) => {
+                match plausible_gpu_counter(
+                    b.saturating_sub(a),
+                    t1.saturating_duration_since(t0),
+                    &gpu.draws_mw,
+                    gpu.ceiling_mw,
+                ) {
+                    GpuPlausibility::Plausible { .. } => CounterStatus::Live,
+                    GpuPlausibility::CounterOverRate { .. } => CounterStatus::Implausible,
+                    GpuPlausibility::BoardTelemetryUnusable { .. } => CounterStatus::Unjudgeable,
                 }
-                _ => true,
-            };
-            if plausible { CounterStatus::Live } else { CounterStatus::Implausible }
-        }
+            }
+            // No pair of samples to compute a rate from. Not a judgement that
+            // the counter is sound; liveness already passed, and there is
+            // nothing here to contradict it.
+            _ => CounterStatus::Live,
+        },
         status => status,
     }
 }
@@ -6504,17 +6629,132 @@ mod tests {
     #[test]
     fn a_gpu_counter_must_count_no_faster_than_its_board_draws() {
         let w = Duration::from_millis(500);
-        assert!(!plausible_gpu_counter(59_750, w, &[12_970, 13_520], Some(65_000)), "119.5 W against a 13 W draw");
-        assert!(!plausible_gpu_counter(22_000, w, &[10_400; 6], Some(65_000)), "44 W against a 10.4 W draw");
-        assert!(!plausible_gpu_counter(60_000, w, &[], Some(65_000)), "120 W on a 65 W board");
-        assert!(!plausible_gpu_counter(44_000, w, &[590_000, 10_300], Some(65_000)), "a 590 W draw reading");
-        assert!(plausible_gpu_counter(6_500, w, &[13_000, 13_500], Some(65_000)), "13 W against a 13 W draw");
-        assert!(plausible_gpu_counter(150_000, w, &[280_000; 5], Some(700_000)), "300 W against a 280 W draw");
-        assert!(plausible_gpu_counter(102_500, w, &[100_000; 5], Some(700_000)), "205 W is 2 x 100 W + 5 W");
-        assert!(!plausible_gpu_counter(102_501, w, &[100_000; 5], Some(700_000)));
-        assert!(plausible_gpu_counter(48_750, w, &[], Some(65_000)), "97.5 W is 1.5 x 65 W");
-        assert!(!plausible_gpu_counter(48_751, w, &[], Some(65_000)));
-        assert!(plausible_gpu_counter(1_000_000, w, &[], None), "nothing to judge by");
+        let p = |mj, draws: &[u32], ceil| plausible_gpu_counter(mj, w, draws, ceil);
+
+        assert!(!p(59_750, &[12_970, 13_520], Some(65_000)).plausible(), "119.5 W against a 13 W draw");
+        assert!(!p(22_000, &[10_400; 6], Some(65_000)).plausible(), "44 W against a 10.4 W draw");
+        assert!(!p(60_000, &[], Some(65_000)).plausible(), "120 W on a 65 W board");
+        assert!(!p(44_000, &[590_000, 10_300], Some(65_000)).plausible(), "88 W against a sane 10.3 W reading");
+        assert!(!p(44_000, &[590_000], Some(65_000)).plausible(), "no usable draw reading at all");
+        assert!(p(6_500, &[13_000, 13_500], Some(65_000)).plausible(), "13 W against a 13 W draw");
+        assert!(p(150_000, &[280_000; 5], Some(700_000)).plausible(), "300 W against a 280 W draw");
+        assert!(p(102_500, &[100_000; 5], Some(700_000)).plausible(), "205 W is 2 x 100 W + 5 W");
+        assert!(!p(102_501, &[100_000; 5], Some(700_000)).plausible());
+        assert!(p(48_750, &[], Some(65_000)).plausible(), "97.5 W is 1.5 x 65 W");
+        assert!(!p(48_751, &[], Some(65_000)).plausible());
+        assert!(p(1_000_000, &[], None).plausible(), "nothing to judge by");
+    }
+
+    /// The verdict carries the magnitude, which a `bool` discarded: a counter
+    /// 1.1 times over a bound and one 20 times over were the same answer.
+    #[test]
+    fn the_verdict_says_which_bound_broke_and_by_how_much() {
+        let w = Duration::from_millis(500);
+
+        // The jetson-hub reading: 119.5 W against a mean 13.2 W draw on a 65 W
+        // board. Both bounds break — 119.5 W is over the 97.5 W cap *and* over
+        // 2 x 13.2 W + 5 W — and the power limit is reported, because it is the
+        // harder of the two: a board cannot draw past its limit by
+        // construction, while the mean-draw bound is soft (telemetry lags a
+        // load step). Naming the harder bound is the more informative
+        // diagnosis.
+        match plausible_gpu_counter(59_750, w, &[12_970, 13_520], Some(65_000)) {
+            GpuPlausibility::CounterOverRate { rate_mw, bound_mw, bound } => {
+                assert_eq!(bound, GpuBound::PowerLimit);
+                assert_eq!((rate_mw, bound_mw), (119_500, 97_500), "1.5 x 65 W");
+            }
+            other => panic!("expected an overrun, got {other:?}"),
+        }
+
+        // The mean-draw bound alone: 44 W sits inside the 97.5 W cap but is
+        // well past 2 x 10.4 W + 5 W, and the verdict names that bound and the
+        // factor a bool threw away.
+        match plausible_gpu_counter(22_000, w, &[10_400; 6], Some(65_000)) {
+            GpuPlausibility::CounterOverRate { rate_mw, bound_mw, bound } => {
+                assert_eq!(bound, GpuBound::MeanDraw);
+                assert_eq!((rate_mw, bound_mw), (44_000, 2 * 10_400 + 5_000));
+                assert!(rate_mw < 97_500, "inside the power cap, so that bound did not bind");
+                assert!(rate_mw / bound_mw >= 1, "{rate_mw} mW over a {bound_mw} mW bound");
+            }
+            other => panic!("expected a mean-draw overrun, got {other:?}"),
+        }
+
+        // With no draw readings the power limit is the only bound, and the
+        // verdict says so rather than leaving the caller to guess which of the
+        // two rejected it.
+        match plausible_gpu_counter(60_000, w, &[], Some(65_000)) {
+            GpuPlausibility::CounterOverRate { rate_mw, bound_mw, bound } => {
+                assert_eq!(bound, GpuBound::PowerLimit);
+                assert_eq!((rate_mw, bound_mw), (120_000, 97_500), "1.5 x 65 W");
+            }
+            other => panic!("expected a power-limit overrun, got {other:?}"),
+        }
+
+        // A marginal overrun is the same status and a different finding: one
+        // milliwatt over, which is what the magnitude is for.
+        match plausible_gpu_counter(48_751, w, &[], Some(65_000)) {
+            GpuPlausibility::CounterOverRate { rate_mw, bound_mw, .. } => {
+                assert_eq!(rate_mw - bound_mw, 2, "just over the cap, not 20 times over");
+            }
+            other => panic!("expected an overrun, got {other:?}"),
+        }
+
+        // And a pass carries the rate too, so a figure near a bound is visible
+        // before it crosses one.
+        assert_eq!(
+            plausible_gpu_counter(6_500, w, &[13_000, 13_500], Some(65_000)),
+            GpuPlausibility::Plausible { rate_mw: 13_000 }
+        );
+    }
+
+    /// A board reporting past its own power limit leaves no yardstick. That is
+    /// an indeterminate, not a finding about the counter — the old code called
+    /// it `Implausible`, whose doc then asserted "it is not counting the board's
+    /// energy", a conclusion this case does not support.
+    #[test]
+    fn an_unusable_board_reading_is_an_indeterminate_not_an_indictment() {
+        let w = Duration::from_millis(500);
+        // A spike beside a sane reading drops the spike, not the judgement.
+        // The sane 10.3 W reading still bounds the counter, so an 88 W counter
+        // is an overrun against 2 x 10.3 W + 5 W — not an indeterminate. The
+        // hardware test on jetson-hub is what forced this: one spike among
+        // otherwise sane 14.8 W samples used to turn a counter running
+        // persistently at 4-30x the board's draw into "cannot judge".
+        match plausible_gpu_counter(44_000, w, &[590_000, 10_300], Some(65_000)) {
+            GpuPlausibility::CounterOverRate { rate_mw, bound_mw, bound } => {
+                assert_eq!(bound, GpuBound::MeanDraw);
+                assert_eq!((rate_mw, bound_mw), (88_000, 2 * 10_300 + 5_000));
+            }
+            other => panic!("one bad sample voided the bracket: {other:?}"),
+        }
+
+        // Dropping the spike also keeps the mean honest. Summing every reading
+        // put the mean at 300 W and the bound at 605 W, which would have passed
+        // an 88 W counter that the sane reading alone refuses.
+        let poisoned_mean = (590_000 + 10_300) / 2;
+        assert!(
+            44_000u64 * 2 < (2 * poisoned_mean + 5_000),
+            "premise: the spike-inflated mean would have admitted this counter"
+        );
+
+        // Only when nothing usable is left is there no yardstick at all.
+        match plausible_gpu_counter(44_000, w, &[590_000], Some(65_000)) {
+            GpuPlausibility::BoardTelemetryUnusable { draw_mw, limit_mw } => {
+                assert_eq!((draw_mw, limit_mw), (590_000, 97_500));
+            }
+            other => panic!("expected an indeterminate, got {other:?}"),
+        }
+        assert!(!plausible_gpu_counter(44_000, w, &[590_000], Some(65_000)).plausible());
+
+        // And the power limit still binds first: it is a static spec value, not
+        // a reading, so a broken draw sample cannot excuse a counter that
+        // overran the board's limit outright.
+        match plausible_gpu_counter(500_000, w, &[590_000], Some(65_000)) {
+            GpuPlausibility::CounterOverRate { bound, bound_mw, .. } => {
+                assert_eq!((bound, bound_mw), (GpuBound::PowerLimit, 97_500));
+            }
+            other => panic!("expected a power-limit overrun, got {other:?}"),
+        }
     }
 
     /// The NVML probe's status: a live counter that ran past its board is
@@ -6542,6 +6782,37 @@ mod tests {
         }
         assert_eq!(nvml_status_of(Some(&live), &big), CounterStatus::Live, "120 W against a 115 W draw on a 700 W board");
         assert_eq!(nvml_status_of(Some(&live), &GpuWatch::default()), CounterStatus::Live, "nothing to judge by");
+
+        // One spike among sane readings is still `Implausible`: the spike is
+        // dropped and the sane readings bound the counter, which overran them.
+        // This is what the jetson-hub hardware test demanded — the first shape
+        // of this code reported the indeterminate here and so disagreed with
+        // the oracle on a counter whose overrun is persistent and repeatable.
+        let mut spiking = GpuWatch { ceiling_mw: Some(65_000), ..GpuWatch::default() };
+        for k in 0..6u32 {
+            let draw = if k == 0 { 590_000 } else { 13_000 };
+            spiking.saw(
+                Some(1_000_000 + 12_000 * u64::from(k)),
+                Some(draw),
+                t0 + Duration::from_millis(100) * k,
+            );
+        }
+        assert_eq!(nvml_status_of(Some(&live), &spiking), CounterStatus::Implausible);
+
+        // `Unjudgeable` is for a bracket with no usable reading left at all: a
+        // board spiking through every sample offers no yardstick. A 60 W
+        // counter on a 65 W board is inside the power cap, so nothing else
+        // can judge it either.
+        let mut all_spikes = GpuWatch { ceiling_mw: Some(65_000), ..GpuWatch::default() };
+        for k in 0..6u32 {
+            all_spikes.saw(
+                Some(1_000_000 + 6_000 * u64::from(k)),
+                Some(590_000),
+                t0 + Duration::from_millis(100) * k,
+            );
+        }
+        assert_eq!(nvml_status_of(Some(&live), &all_spikes), CounterStatus::Unjudgeable);
+        assert!(!admits_counter(CounterStatus::Unjudgeable), "unjudgeable is not admissible");
     }
 
     /// The tests' interval for a CPU-counter probe: a 25 ms tick and a short
@@ -6808,7 +7079,13 @@ mod tests {
         #[cfg(target_os = "macos")]
         let _hw = ioreport_hw_lock();
         let fake = FakePowercap::new(&["1000"]);
-        for status in [CounterStatus::Unavailable, CounterStatus::Frozen, CounterStatus::Implausible, CounterStatus::Live] {
+        for status in [
+            CounterStatus::Unavailable,
+            CounterStatus::Frozen,
+            CounterStatus::Implausible,
+            CounterStatus::Unjudgeable,
+            CounterStatus::Live,
+        ] {
             let opened = Cell::new(0);
             let open = || {
                 opened.set(opened.get() + 1);