diff --git a/jcp/SPEC.md b/jcp/SPEC.md index ee40a7a2a..b20e80943 100644 --- a/jcp/SPEC.md +++ b/jcp/SPEC.md @@ -404,6 +404,46 @@ its estimator or reports the work unmetered, and SHOULD record why. A figure MUST NOT claim coverage of a domain its counter does not read; where the format has a coverage field, such a figure is Partial. +**Plausibility (REQUIRED where a bound is readable).** Liveness is necessary and +not sufficient: a counter can advance continuously and still not be counting +what it claims. An NVML total-energy counter on an RTX 4050 Laptop GPU (driver +595.84) advances in every sample interval while running at four to thirty times +the board's own power draw for as long as it is polled — perfectly live, and no +count of that board's energy. A figure from it would debit energy the device +never spent. + +So where the platform exposes a bound that is read **independently of the +counter** — the board's maximum power limit, the board's instantaneous power +draw — a hardware-graded figure's implied rate MUST be checked against it, and a +counter whose rate exceeds the bound MUST NOT produce a hardware-graded figure. +An implementation SHOULD record the rate and the bound it broke, not merely that +it broke one: a counter a tenth over a bound and one twenty times over are the +same refusal and not the same finding, and re-grading a past refusal against a +revised bound is only possible if the rate survived the check. + +A bound reading that is itself impossible is not a reading: the same RTX 4050 +reports about 590 W, past its own 65 W limit, for a second after it wakes. Such a +reading MUST be discarded — and only that reading. One broken sample is not a +broken bracket, and discarding the whole judgement on it reports an indeterminate +where the evidence supports a finding: on the reference hardware a single spike +among otherwise sane 14.8 W samples would otherwise excuse a counter running +persistently at four to thirty times the board's draw. Discarding them +individually also keeps the comparison honest, since a spike left in a mean makes +the bound it derives *more* permissive. + +Only where **no usable reading remains** is the counter **unjudgeable** rather +than implausible: there is then no yardstick, so there is no finding about the +counter. It MUST NOT produce a hardware-graded figure either, because unjudgeable +is not admissible and declining is the safe direction, but an implementation MUST +NOT report it as a counter that overran. The two call for different operator +responses — replacing a counter, or waiting for a board to settle — and +conflating them loses that. A bound that is a static limit rather than a reading +(the board's maximum power limit) is never discarded, so a broken draw sample +cannot excuse a counter that overran that limit outright. + +Where no such bound is readable, this requirement does not apply; liveness +stands alone. + The record of why is the implementation's to choose; the reference meter exposes each counter's liveness status, and which rung it selected with the status of every rung above it that it declined. Coverage diff --git a/jcp/jcp-rs/crates/jcp-energy/src/lib.rs b/jcp/jcp-rs/crates/jcp-energy/src/lib.rs index 930183d30..a4e90385a 100644 --- a/jcp/jcp-rs/crates/jcp-energy/src/lib.rs +++ b/jcp/jcp-rs/crates/jcp-energy/src/lib.rs @@ -646,13 +646,30 @@ pub enum CounterStatus { Frozen, /// The counter kept advancing, but a GPU counter (NVML's) ran faster than /// its board can draw power over the probe: past 1.5 times the board's - /// maximum power limit, past twice the board's own power draw plus 5 W, - /// or with the board's draw reading itself past that limit. It is not - /// counting the board's energy, so a figure from it would debit energy - /// the device never spent. Seen on an RTX 4050 Laptop GPU (driver + /// maximum power limit, or past twice the board's own power draw plus 5 W. + /// It is not counting the board's energy, so a figure from it would debit + /// energy the device never spent. Seen on an RTX 4050 Laptop GPU (driver /// 595.84), whose counter runs at 4 to 30 times the board's draw while it /// is polled. Never selected. + /// + /// By how much it overran is in [`GpuPlausibility::CounterOverRate`], which + /// carries the rate and the bound it broke: a counter 1.1 times over a + /// bound and one 20 times over are the same status but not the same + /// finding, and tuning the margins later needs the figures. Implausible, + /// The counter kept advancing and could not be judged, because the board's + /// own power telemetry was unusable over the probe: a draw reading past the + /// board's own maximum power limit (an RTX 4050 Laptop GPU reports about + /// 590 W for a second after it wakes). + /// + /// Split out from [`CounterStatus::Implausible`], which used to cover this + /// and whose doc then asserted "it is not counting the board's energy" — + /// a conclusion this case does not support. The yardstick is broken, not + /// necessarily the counter. Never selected either way: unjudgeable is not + /// admissible, and erring toward declining is the safe direction. The + /// distinction is the diagnosis an operator gets, which is the difference + /// between replacing a counter and waiting for the board to settle. + Unjudgeable, /// Every rail of the counter kept advancing while the probe ran: each in /// at least five separate sample intervals, making up at least 80% of /// those observed, over at least 300 ms. The rung is selected. @@ -855,23 +872,122 @@ const GPU_DRAW_SLACK_MW: u128 = 5_000; /// 595.84) the total-energy counter runs several times faster than the /// board draws for as long as it is polled: at this probe's 100 ms wait it /// ran at 58-305 W while the board drew 5-14 W against a 65 W limit. -fn plausible_gpu_counter(energy_mj: u64, window: std::time::Duration, draws_mw: &[u32], ceiling_mw: Option) -> bool { +/// Which bound a GPU counter's rate broke. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum GpuBound { + /// [`GPU_CEILING_MARGIN`] of the board's maximum power limit. + PowerLimit, + /// [`GPU_DRAW_FACTOR`] times the board's mean draw plus + /// [`GPU_DRAW_SLACK_MW`]. + MeanDraw, +} + +/// What the plausibility check found, with the figures it found it from. +/// +/// A `bool` was the whole of this, and it threw away everything an operator or +/// an auditor needs: which of the bounds broke, by how much, and the rate it +/// broke them at. A counter 1.1 times over a bound and one 20 times over were +/// the same answer. The figures also make the judgement reproducible — the +/// margins are constants in this crate, and re-grading a past finding against +/// a different margin is only possible if the rate survived the check. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum GpuPlausibility { + /// The counter's rate sits inside every bound whose reading was present. + Plausible { + /// The rate the counter implies over the window, in milliwatts. + rate_mw: u64, + }, + /// The counter ran faster than the board can draw. + CounterOverRate { + /// The rate the counter implies over the window, in milliwatts. + rate_mw: u64, + /// The bound it broke, in milliwatts. + bound_mw: u64, + /// Which bound that was. + bound: GpuBound, + }, + /// The board's own telemetry read past the board's own power limit, so + /// there is no yardstick to judge the counter against. An indeterminate, + /// not a finding about the counter — see [`CounterStatus::Unjudgeable`]. + BoardTelemetryUnusable { + /// The offending draw reading, in milliwatts. + draw_mw: u64, + /// The limit it exceeded (the margin-adjusted cap), in milliwatts. + limit_mw: u64, + }, +} + +impl GpuPlausibility { + /// Whether the counter was judged to be counting its board's energy. + /// + /// `false` for an indeterminate as well as for an overrun: unjudgeable is + /// not admissible, and erring toward declining is the safe direction. Use + /// the variant itself to tell the two apart. + pub fn plausible(&self) -> bool { + matches!(self, GpuPlausibility::Plausible { .. }) + } +} + +fn plausible_gpu_counter( + energy_mj: u64, + window: std::time::Duration, + draws_mw: &[u32], + ceiling_mw: Option, +) -> GpuPlausibility { let ns = window.as_nanos().max(1); // mJ per second is mW. let rate_mw = u128::from(energy_mj) * 1_000_000_000 / ns; - if let Some(limit) = ceiling_mw.filter(|&l| l > 0) { - let cap = u128::from(limit) * GPU_CEILING_MARGIN.0 / GPU_CEILING_MARGIN.1; - if rate_mw > cap || draws_mw.iter().any(|&d| u128::from(d) > cap) { - return false; + let clamp = |v: u128| u64::try_from(v).unwrap_or(u64::MAX); + let cap = ceiling_mw + .filter(|&l| l > 0) + .map(|l| u128::from(l) * GPU_CEILING_MARGIN.0 / GPU_CEILING_MARGIN.1); + + // A draw reading past the board's own limit is not a reading: an RTX 4050 + // Laptop GPU reports about 590 W for a second after it wakes. Such a + // reading is dropped, and only that reading — it is one broken sample, not + // a broken bracket. Voiding the whole judgement on it was the first shape + // of this code and the hardware test refused it: on jetson-hub one spike + // among otherwise sane 14.8 W samples turned a counter running + // persistently at 4-30x the board's draw into an indeterminate, which is a + // worse diagnosis than the overrun it actually is. + // + // Dropping them also keeps the mean honest. The old code summed every + // reading, so a 590 W spike inflated the mean and made the mean-draw bound + // *more* permissive — it only ever went unnoticed because any such reading + // short-circuited the whole check first. + let (usable, unusable): (Vec, Vec) = draws_mw + .iter() + .map(|&d| u128::from(d)) + .partition(|&d| cap.is_none_or(|c| d <= c)); + + if let Some(cap) = cap { + if rate_mw > cap { + return GpuPlausibility::CounterOverRate { + rate_mw: clamp(rate_mw), + bound_mw: clamp(cap), + bound: GpuBound::PowerLimit, + }; + } + // Only when nothing usable is left is there no yardstick at all. + if usable.is_empty() && !unusable.is_empty() { + return GpuPlausibility::BoardTelemetryUnusable { + draw_mw: clamp(unusable[0]), + limit_mw: clamp(cap), + }; } } - if !draws_mw.is_empty() { - let mean = draws_mw.iter().map(|&d| u128::from(d)).sum::() / draws_mw.len() as u128; - if rate_mw > GPU_DRAW_FACTOR * mean + GPU_DRAW_SLACK_MW { - return false; + if !usable.is_empty() { + let mean = usable.iter().sum::() / usable.len() as u128; + let bound = GPU_DRAW_FACTOR * mean + GPU_DRAW_SLACK_MW; + if rate_mw > bound { + return GpuPlausibility::CounterOverRate { + rate_mw: clamp(rate_mw), + bound_mw: clamp(bound), + bound: GpuBound::MeanDraw, + }; } } - true + GpuPlausibility::Plausible { rate_mw: clamp(rate_mw) } } /// The status an NVML probe reports: [`liveness::status_of`] its run, except @@ -880,15 +996,24 @@ fn plausible_gpu_counter(energy_mj: u64, window: std::time::Duration, draws_mw: #[cfg_attr(not(any(test, feature = "nvml")), allow(dead_code))] fn nvml_status_of(probed: Option<&liveness::Advances>, gpu: &GpuWatch) -> CounterStatus { match liveness::status_of(probed) { - CounterStatus::Live => { - let plausible = match (gpu.first, gpu.last) { - (Some((a, t0)), Some((b, t1))) => { - plausible_gpu_counter(b.saturating_sub(a), t1.saturating_duration_since(t0), &gpu.draws_mw, gpu.ceiling_mw) + CounterStatus::Live => match (gpu.first, gpu.last) { + (Some((a, t0)), Some((b, t1))) => { + match plausible_gpu_counter( + b.saturating_sub(a), + t1.saturating_duration_since(t0), + &gpu.draws_mw, + gpu.ceiling_mw, + ) { + GpuPlausibility::Plausible { .. } => CounterStatus::Live, + GpuPlausibility::CounterOverRate { .. } => CounterStatus::Implausible, + GpuPlausibility::BoardTelemetryUnusable { .. } => CounterStatus::Unjudgeable, } - _ => true, - }; - if plausible { CounterStatus::Live } else { CounterStatus::Implausible } - } + } + // No pair of samples to compute a rate from. Not a judgement that + // the counter is sound; liveness already passed, and there is + // nothing here to contradict it. + _ => CounterStatus::Live, + }, status => status, } } @@ -6504,17 +6629,132 @@ mod tests { #[test] fn a_gpu_counter_must_count_no_faster_than_its_board_draws() { let w = Duration::from_millis(500); - assert!(!plausible_gpu_counter(59_750, w, &[12_970, 13_520], Some(65_000)), "119.5 W against a 13 W draw"); - assert!(!plausible_gpu_counter(22_000, w, &[10_400; 6], Some(65_000)), "44 W against a 10.4 W draw"); - assert!(!plausible_gpu_counter(60_000, w, &[], Some(65_000)), "120 W on a 65 W board"); - assert!(!plausible_gpu_counter(44_000, w, &[590_000, 10_300], Some(65_000)), "a 590 W draw reading"); - assert!(plausible_gpu_counter(6_500, w, &[13_000, 13_500], Some(65_000)), "13 W against a 13 W draw"); - assert!(plausible_gpu_counter(150_000, w, &[280_000; 5], Some(700_000)), "300 W against a 280 W draw"); - assert!(plausible_gpu_counter(102_500, w, &[100_000; 5], Some(700_000)), "205 W is 2 x 100 W + 5 W"); - assert!(!plausible_gpu_counter(102_501, w, &[100_000; 5], Some(700_000))); - assert!(plausible_gpu_counter(48_750, w, &[], Some(65_000)), "97.5 W is 1.5 x 65 W"); - assert!(!plausible_gpu_counter(48_751, w, &[], Some(65_000))); - assert!(plausible_gpu_counter(1_000_000, w, &[], None), "nothing to judge by"); + let p = |mj, draws: &[u32], ceil| plausible_gpu_counter(mj, w, draws, ceil); + + assert!(!p(59_750, &[12_970, 13_520], Some(65_000)).plausible(), "119.5 W against a 13 W draw"); + assert!(!p(22_000, &[10_400; 6], Some(65_000)).plausible(), "44 W against a 10.4 W draw"); + assert!(!p(60_000, &[], Some(65_000)).plausible(), "120 W on a 65 W board"); + assert!(!p(44_000, &[590_000, 10_300], Some(65_000)).plausible(), "88 W against a sane 10.3 W reading"); + assert!(!p(44_000, &[590_000], Some(65_000)).plausible(), "no usable draw reading at all"); + assert!(p(6_500, &[13_000, 13_500], Some(65_000)).plausible(), "13 W against a 13 W draw"); + assert!(p(150_000, &[280_000; 5], Some(700_000)).plausible(), "300 W against a 280 W draw"); + assert!(p(102_500, &[100_000; 5], Some(700_000)).plausible(), "205 W is 2 x 100 W + 5 W"); + assert!(!p(102_501, &[100_000; 5], Some(700_000)).plausible()); + assert!(p(48_750, &[], Some(65_000)).plausible(), "97.5 W is 1.5 x 65 W"); + assert!(!p(48_751, &[], Some(65_000)).plausible()); + assert!(p(1_000_000, &[], None).plausible(), "nothing to judge by"); + } + + /// The verdict carries the magnitude, which a `bool` discarded: a counter + /// 1.1 times over a bound and one 20 times over were the same answer. + #[test] + fn the_verdict_says_which_bound_broke_and_by_how_much() { + let w = Duration::from_millis(500); + + // The jetson-hub reading: 119.5 W against a mean 13.2 W draw on a 65 W + // board. Both bounds break — 119.5 W is over the 97.5 W cap *and* over + // 2 x 13.2 W + 5 W — and the power limit is reported, because it is the + // harder of the two: a board cannot draw past its limit by + // construction, while the mean-draw bound is soft (telemetry lags a + // load step). Naming the harder bound is the more informative + // diagnosis. + match plausible_gpu_counter(59_750, w, &[12_970, 13_520], Some(65_000)) { + GpuPlausibility::CounterOverRate { rate_mw, bound_mw, bound } => { + assert_eq!(bound, GpuBound::PowerLimit); + assert_eq!((rate_mw, bound_mw), (119_500, 97_500), "1.5 x 65 W"); + } + other => panic!("expected an overrun, got {other:?}"), + } + + // The mean-draw bound alone: 44 W sits inside the 97.5 W cap but is + // well past 2 x 10.4 W + 5 W, and the verdict names that bound and the + // factor a bool threw away. + match plausible_gpu_counter(22_000, w, &[10_400; 6], Some(65_000)) { + GpuPlausibility::CounterOverRate { rate_mw, bound_mw, bound } => { + assert_eq!(bound, GpuBound::MeanDraw); + assert_eq!((rate_mw, bound_mw), (44_000, 2 * 10_400 + 5_000)); + assert!(rate_mw < 97_500, "inside the power cap, so that bound did not bind"); + assert!(rate_mw / bound_mw >= 1, "{rate_mw} mW over a {bound_mw} mW bound"); + } + other => panic!("expected a mean-draw overrun, got {other:?}"), + } + + // With no draw readings the power limit is the only bound, and the + // verdict says so rather than leaving the caller to guess which of the + // two rejected it. + match plausible_gpu_counter(60_000, w, &[], Some(65_000)) { + GpuPlausibility::CounterOverRate { rate_mw, bound_mw, bound } => { + assert_eq!(bound, GpuBound::PowerLimit); + assert_eq!((rate_mw, bound_mw), (120_000, 97_500), "1.5 x 65 W"); + } + other => panic!("expected a power-limit overrun, got {other:?}"), + } + + // A marginal overrun is the same status and a different finding: one + // milliwatt over, which is what the magnitude is for. + match plausible_gpu_counter(48_751, w, &[], Some(65_000)) { + GpuPlausibility::CounterOverRate { rate_mw, bound_mw, .. } => { + assert_eq!(rate_mw - bound_mw, 2, "just over the cap, not 20 times over"); + } + other => panic!("expected an overrun, got {other:?}"), + } + + // And a pass carries the rate too, so a figure near a bound is visible + // before it crosses one. + assert_eq!( + plausible_gpu_counter(6_500, w, &[13_000, 13_500], Some(65_000)), + GpuPlausibility::Plausible { rate_mw: 13_000 } + ); + } + + /// A board reporting past its own power limit leaves no yardstick. That is + /// an indeterminate, not a finding about the counter — the old code called + /// it `Implausible`, whose doc then asserted "it is not counting the board's + /// energy", a conclusion this case does not support. + #[test] + fn an_unusable_board_reading_is_an_indeterminate_not_an_indictment() { + let w = Duration::from_millis(500); + // A spike beside a sane reading drops the spike, not the judgement. + // The sane 10.3 W reading still bounds the counter, so an 88 W counter + // is an overrun against 2 x 10.3 W + 5 W — not an indeterminate. The + // hardware test on jetson-hub is what forced this: one spike among + // otherwise sane 14.8 W samples used to turn a counter running + // persistently at 4-30x the board's draw into "cannot judge". + match plausible_gpu_counter(44_000, w, &[590_000, 10_300], Some(65_000)) { + GpuPlausibility::CounterOverRate { rate_mw, bound_mw, bound } => { + assert_eq!(bound, GpuBound::MeanDraw); + assert_eq!((rate_mw, bound_mw), (88_000, 2 * 10_300 + 5_000)); + } + other => panic!("one bad sample voided the bracket: {other:?}"), + } + + // Dropping the spike also keeps the mean honest. Summing every reading + // put the mean at 300 W and the bound at 605 W, which would have passed + // an 88 W counter that the sane reading alone refuses. + let poisoned_mean = (590_000 + 10_300) / 2; + assert!( + 44_000u64 * 2 < (2 * poisoned_mean + 5_000), + "premise: the spike-inflated mean would have admitted this counter" + ); + + // Only when nothing usable is left is there no yardstick at all. + match plausible_gpu_counter(44_000, w, &[590_000], Some(65_000)) { + GpuPlausibility::BoardTelemetryUnusable { draw_mw, limit_mw } => { + assert_eq!((draw_mw, limit_mw), (590_000, 97_500)); + } + other => panic!("expected an indeterminate, got {other:?}"), + } + assert!(!plausible_gpu_counter(44_000, w, &[590_000], Some(65_000)).plausible()); + + // And the power limit still binds first: it is a static spec value, not + // a reading, so a broken draw sample cannot excuse a counter that + // overran the board's limit outright. + match plausible_gpu_counter(500_000, w, &[590_000], Some(65_000)) { + GpuPlausibility::CounterOverRate { bound, bound_mw, .. } => { + assert_eq!((bound, bound_mw), (GpuBound::PowerLimit, 97_500)); + } + other => panic!("expected a power-limit overrun, got {other:?}"), + } } /// The NVML probe's status: a live counter that ran past its board is @@ -6542,6 +6782,37 @@ mod tests { } assert_eq!(nvml_status_of(Some(&live), &big), CounterStatus::Live, "120 W against a 115 W draw on a 700 W board"); assert_eq!(nvml_status_of(Some(&live), &GpuWatch::default()), CounterStatus::Live, "nothing to judge by"); + + // One spike among sane readings is still `Implausible`: the spike is + // dropped and the sane readings bound the counter, which overran them. + // This is what the jetson-hub hardware test demanded — the first shape + // of this code reported the indeterminate here and so disagreed with + // the oracle on a counter whose overrun is persistent and repeatable. + let mut spiking = GpuWatch { ceiling_mw: Some(65_000), ..GpuWatch::default() }; + for k in 0..6u32 { + let draw = if k == 0 { 590_000 } else { 13_000 }; + spiking.saw( + Some(1_000_000 + 12_000 * u64::from(k)), + Some(draw), + t0 + Duration::from_millis(100) * k, + ); + } + assert_eq!(nvml_status_of(Some(&live), &spiking), CounterStatus::Implausible); + + // `Unjudgeable` is for a bracket with no usable reading left at all: a + // board spiking through every sample offers no yardstick. A 60 W + // counter on a 65 W board is inside the power cap, so nothing else + // can judge it either. + let mut all_spikes = GpuWatch { ceiling_mw: Some(65_000), ..GpuWatch::default() }; + for k in 0..6u32 { + all_spikes.saw( + Some(1_000_000 + 6_000 * u64::from(k)), + Some(590_000), + t0 + Duration::from_millis(100) * k, + ); + } + assert_eq!(nvml_status_of(Some(&live), &all_spikes), CounterStatus::Unjudgeable); + assert!(!admits_counter(CounterStatus::Unjudgeable), "unjudgeable is not admissible"); } /// The tests' interval for a CPU-counter probe: a 25 ms tick and a short @@ -6808,7 +7079,13 @@ mod tests { #[cfg(target_os = "macos")] let _hw = ioreport_hw_lock(); let fake = FakePowercap::new(&["1000"]); - for status in [CounterStatus::Unavailable, CounterStatus::Frozen, CounterStatus::Implausible, CounterStatus::Live] { + for status in [ + CounterStatus::Unavailable, + CounterStatus::Frozen, + CounterStatus::Implausible, + CounterStatus::Unjudgeable, + CounterStatus::Live, + ] { let opened = Cell::new(0); let open = || { opened.set(opened.get() + 1);