4d745abf6b
Some checks failed
energy meters on real hardware / RAPL and NVML on jetson-hub (pull_request) Successful in 44s
standards workspaces build + test / which workspaces this change touches (pull_request) Successful in 20s
repository checks / lockfiles, tests in CI, workflow hygiene, spec and section references, public text (pull_request) Successful in 1m47s
standards workspaces build + test / macOS energy backends (pull_request) Failing after 2m42s
standards workspaces build + test / eoc-rs (eoc/eoc-rs, -p eoc-meter --features cuda, eoc-rs, linux) (pull_request) Successful in 2m17s
The GPU-freshness report left a comment explaining why a rise the oracle saw is not one the probe must have seen. It did not say what still covers the property that assertion was reaching for, so the obvious next reader restores a hardware assertion and reintroduces the flake. It is covered deterministically, per sample, by the scripted probes: `probe_with_a_single_batch_jump_is_frozen` drives `GPU Energy` to a different value at every sample and asserts each one is exactly what the rule was handed, and `probe_with_one_die_live_and_one_frozen_is_frozen` does the same. A sampler that cached one reading fails those on sample 1, on every host, with no dependence on what the hardware happens to do. So the earlier claim that the report gave up that coverage was wrong: it gave up nothing, and driving GPU work inside the probe's window to restore a hardware assertion is unnecessary. The comment now says so, and says why no sound hardware witness exists on the CI host anyway -- its CPU rails are frozen and its GPU rail is activity-driven. Comment only; no behaviour change. Co-Authored-By: Claude Opus 5 <[email protected]>
5804 lines
288 KiB
Rust
5804 lines
288 KiB
Rust
//! EOC joule meter.
|
||
//!
|
||
//! The meter reads cumulative micro-joule counters from whatever hardware
|
||
//! is available. Production deployments care a great deal which counter
|
||
//! is in use; reference deployments fall back to `StubCounter`, which
|
||
//! always reads 0.
|
||
//!
|
||
//! A counter carries no provenance of its own: [`JouleCounter`] only reads,
|
||
//! and its consumers decide. eoc-cascade, eoc-streaming and eoc-local's
|
||
//! `mlx`, `mlc` and `onnx` backends call a delta `JouleSource::Measured`
|
||
//! only when the counter advanced over the span they read it across.
|
||
//! Anything else — a failed read, a counter that did not move, the stub,
|
||
//! which never moves — is no reading, and yields no measured figure:
|
||
//! eoc-streaming reports none, eoc-cascade keeps the answering stage's own
|
||
//! cost as the stage reported it, and those backends report an estimate.
|
||
//! eoc-cascade reads a span only around its neural stage: a hit on its
|
||
//! cache, KV stage or graph is shorter than a read of a counter, so it is
|
||
//! charged what the counter read over a batch of such hits in one span, per
|
||
//! hit, and that is an estimate.
|
||
//!
|
||
//! Hardware counter dependencies are **platform-conditional and
|
||
//! feature-gated**:
|
||
//!
|
||
//! Every hardware counter is held to one liveness rule before [`detect`]
|
||
//! selects it: probed once per process, it must keep advancing — in at least
|
||
//! five separate sample intervals, at least 80% of those observed, over at
|
||
//! least 300 ms. A counter that holds, or is republished in lumps, is
|
||
//! declined, because a lump landing in one span would be read as that span's
|
||
//! energy. A live counter's delta is a reading; nothing else is.
|
||
//!
|
||
//! - Linux RAPL is read through powercap (`/sys/class/powercap`) as the sum
|
||
//! of every package zone — each top-level `intel-rapl:<n>` zone whose `name`
|
||
//! reads `package-<n>`, so every socket of a multi-socket server, and never
|
||
//! `psys` (the platform zone, whose energy includes the packages'), `dram`
|
||
//! or a sub-zone — each zone's wrap taken at its own `max_energy_range_uj`;
|
||
//! **only when every package counter reads and is live** ([`rapl_status`];
|
||
//! the probe burns CPU on the calling thread). No external crate required.
|
||
//! - Apple Silicon IOReport Energy Model counters are read through Apple's
|
||
//! IOReport C API (no sudo, no external binary) — **only when every CPU
|
||
//! energy rail is live**, each die's on a multi-die part
|
||
//! ([`ioreport_status`]). A subscription can open while every CPU rail
|
||
//! holds its value, refreshed only now and then; the summed rails then
|
||
//! advance through the GPU rail, a stale refresh, or not at all, and would
|
||
//! pass that off as a reading of CPU work. The counter sums every die's CPU
|
||
//! rail, so one die live beside one frozen is not live either. When a CPU
|
||
//! rail is not live the counter is declined and [`detect`] falls through to
|
||
//! the next rung. A live IOReport counter is a vendor energy counter, as
|
||
//! RAPL and NVML are, so a delta it reads is `Measured` — but only while the
|
||
//! rails are live: the counter exists only on a host whose rails the probe
|
||
//! found live, and on any other there is no IOReport reading to tag.
|
||
//! - macOS per-process energy ([`MacosProcessEnergyCounter`]): this
|
||
//! process's own threads' CPU energy, from `proc_pid_rusage`, live for an
|
||
//! unprivileged reader. **Never selected by [`detect`]** — it does not see
|
||
//! a child process's work, or the GPU's — only by [`detect_in_process`],
|
||
//! for callers whose metered work runs on the CPU in this process. The
|
||
//! kernel adds a thread's energy to the process's count only when the
|
||
//! thread leaves a core, so **each read is taken only once the reading
|
||
//! thread is seen to have left its core** (it sleeps at least 50 µs
|
||
//! first, and checks); a read whose thread was not seen to leave fails,
|
||
//! and its span is no reading: the caller keeps its estimate.
|
||
//! - macOS `powermetrics`: detected when the binary is on the PATH, but the
|
||
//! reference wrapper does not parse it, so every read returns
|
||
//! `Err(NoCounter)`: no figure, and callers keep their own estimate.
|
||
//! - NVIDIA NVML is gated behind the `cuda` feature (off by default), and
|
||
//! **never selected by [`detect`]**: [`NvmlCounter`] reads GPU 0's
|
||
//! total-energy counter, which is the GPU's energy and not the work's (the
|
||
//! CPU's share of it goes unread), and eoc's consumers grade any advance of
|
||
//! the counter they are handed `Measured`, with no coverage field to say a
|
||
//! figure covers part of the work. By the family's rule a figure that does
|
||
//! not cover the work it is booked to is an estimate in such a format, so
|
||
//! the GPU counter is not a [`JouleCounter`]: it is GPU telemetry, read
|
||
//! beside one. It is constructed only while its counter is live and
|
||
//! plausibly counts the board's energy ([`nvml_status`],
|
||
//! [`CounterStatus::Implausible`]).
|
||
//! - WASM and unknown targets fall through to `StubCounter`.
|
||
|
||
// `deny` not `forbid`: the whole crate is safe except three audited FFI
|
||
// modules, each with a scoped `#[allow(unsafe_code)]`: the Apple-Silicon
|
||
// IOReport module (`macos_ioreport`) — the on-die energy counters are
|
||
// unreachable without the `unsafe extern "C"` IOReport calls — `clocks`, whose
|
||
// one call is `clock_gettime` for the probing thread's CPU clock, and
|
||
// `macos_process_energy`, whose calls are `getpid`, `proc_pid_rusage`, and,
|
||
// for the reading thread's own energy count, `pthread_self`,
|
||
// `pthread_threadid_np` and `proc_pidinfo`.
|
||
// Everything else stays unsafe-free.
|
||
#![deny(unsafe_code)]
|
||
|
||
// `Error` is used by platform-conditional modules below.
|
||
#[cfg(any(target_os = "linux", target_os = "macos", feature = "cuda"))]
|
||
use eoc_core::Error;
|
||
use eoc_core::Result;
|
||
|
||
/// A monotonic, cumulative joule counter.
|
||
///
|
||
/// Implementations report total energy consumed since boot in micro-joules.
|
||
/// Subtract two readings to get the energy of an interval.
|
||
pub trait JouleCounter: Send + Sync {
|
||
/// Read the current cumulative micro-joule count.
|
||
fn read_microjoules(&self) -> Result<u64>;
|
||
|
||
/// Human-readable name of this counter (`"rapl"`, `"nvml"`, etc.).
|
||
fn name(&self) -> &'static str;
|
||
}
|
||
|
||
/// A no-op counter that always reports 0. Used when no hardware counter
|
||
/// is available.
|
||
#[derive(Debug, Default)]
|
||
pub struct StubCounter;
|
||
|
||
impl JouleCounter for StubCounter {
|
||
fn read_microjoules(&self) -> Result<u64> {
|
||
Ok(0)
|
||
}
|
||
fn name(&self) -> &'static str {
|
||
"stub"
|
||
}
|
||
}
|
||
|
||
// The sustained-advance liveness rule, kept apart from every counter's FFI so
|
||
// every part of it — the name rule, the pairing of samples, the per-rail count,
|
||
// when to stop sampling, and the verdict's mapping to a status — runs under test
|
||
// with scripted samples and a scripted clock. The FFI only takes a sample and
|
||
// lists its channels as (name, absolute raw integer); everything that decides
|
||
// lives here. One rule for every vendor counter: IOReport watches the CPU energy
|
||
// rails of an Energy Model sample; RAPL hands each package counter as its own
|
||
// channel, and NVML its one counter. Every rail watched must keep advancing, each
|
||
// on its own, because a counter's reading sums every rail it watches: IOReport's
|
||
// adds every die's `CPU Energy`, RAPL's every package's. Compiled on every platform, so a Linux CI leg runs its tests.
|
||
#[cfg_attr(not(target_os = "macos"), allow(dead_code))]
|
||
mod rail_liveness {
|
||
use std::time::Duration;
|
||
|
||
/// In how many separate sample intervals the same CPU rail must advance
|
||
/// before it can count as live; see [`sustained`] for the whole rule.
|
||
///
|
||
/// One is not enough, and neither is two. On macOS 27.0.1 an unprivileged
|
||
/// reader's CPU rails are not permanently frozen; they are refreshed in
|
||
/// rare batches. An M3 Ultra's two dies held for at least 22 minutes, then
|
||
/// stepped +1,344,869 and +1,391,637 mJ within one 100 ms sample, then held
|
||
/// again; an M5 Max's `CPU Energy` did not change in 13,614 samples over 25
|
||
/// minutes, yet had moved from 480,083,160 to 541,893,638 mJ earlier that
|
||
/// hour. And a refresh is not always one step: a watcher sampling the M5
|
||
/// Max's Energy Model group every ~16 ms saw `CPU Energy` step
|
||
/// 2,643,843,378 -> 2,705,607,863 (+61,764,485), then 2,705,607,863 ->
|
||
/// 2,705,611,298 (+3,435) 64 ms later (`MCPU0`, `MCPU1` and `PCPU` each
|
||
/// stepped twice too). The probe samples every ~31 ms, so a probe whose
|
||
/// window holds such a refresh sees two advancing intervals. Called live,
|
||
/// it would select the counter for the life of the process, and every
|
||
/// later refresh would land, whole, in whichever caller's interval
|
||
/// straddled it. A live rail sampled every ~30 ms under load steps in
|
||
/// nearly every interval, for as long as it is watched.
|
||
pub(crate) const MIN_LIVE_ADVANCES: u32 = 5;
|
||
|
||
/// How much wall time, from the first sample, the probe must have watched
|
||
/// a rail advance before it can count as live. A refresh published in
|
||
/// steps lands within ~64 ms; five or more intervals at the probe's
|
||
/// ~31 ms can all fall inside a burst that short, and would then be every
|
||
/// interval observed. Over 300 ms of sustained advance they cannot.
|
||
pub(crate) const MIN_LIVE_WINDOW: Duration = Duration::from_millis(300);
|
||
|
||
/// However slow sampling is, the probe stops after this many budgets.
|
||
pub(crate) const PROBE_BUDGET_CAP: u32 = 5;
|
||
|
||
/// The liveness rule for one rail, evaluated after each interval: it
|
||
/// advanced in at least [`MIN_LIVE_ADVANCES`] separate intervals, those
|
||
/// were at least 80% of every interval observed so far, and the samples
|
||
/// observed so far span at least [`MIN_LIVE_WINDOW`] of wall time from the
|
||
/// first. A burst of steps — a refresh — fails the window while it is
|
||
/// young and the share once it is not; a live rail passes all three.
|
||
pub(crate) fn sustained(advances: u32, intervals: u32, window: Duration) -> bool {
|
||
advances >= MIN_LIVE_ADVANCES
|
||
&& u64::from(advances) * 5 >= u64::from(intervals) * 4
|
||
&& window >= MIN_LIVE_WINDOW
|
||
}
|
||
|
||
/// What the liveness probe saw of the CPU energy rails.
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||
pub(crate) enum CpuRail {
|
||
/// Every CPU energy rail in the samples kept advancing, each by
|
||
/// [`sustained`]: in at least [`MIN_LIVE_ADVANCES`] separate intervals,
|
||
/// at least 80% of those observed, over at least [`MIN_LIVE_WINDOW`].
|
||
Live,
|
||
/// A CPU energy rail exists, but not every one kept advancing: one
|
||
/// held, or stepped only in a burst (a batch refresh, in one step or
|
||
/// several), though another may have been live.
|
||
Frozen,
|
||
/// Samples were taken, and no `CPU Energy` channel was in them.
|
||
Absent,
|
||
}
|
||
|
||
/// `DIE_<n>_CPU Energy` → `CPU Energy`; any other name unchanged.
|
||
pub(crate) fn strip_die_prefix(name: &str) -> &str {
|
||
if let Some(rest) = name.strip_prefix("DIE_") {
|
||
let after = rest.trim_start_matches(|c: char| c.is_ascii_digit());
|
||
if let Some(r) = after.strip_prefix('_') {
|
||
return r;
|
||
}
|
||
}
|
||
name
|
||
}
|
||
|
||
/// A CPU energy aggregate: `CPU Energy`, optionally `DIE_<n>_`-prefixed,
|
||
/// compared case-insensitively. Sub-rails (`PCPU`, `MCPU*`, `PACC_*`,
|
||
/// `DIE_n_PACC*_CPM*`) are not the aggregate and are not watched.
|
||
pub(crate) fn is_cpu_energy_rail(name: &str) -> bool {
|
||
strip_die_prefix(name).eq_ignore_ascii_case("CPU Energy")
|
||
}
|
||
|
||
/// A GPU energy aggregate: `GPU Energy`, optionally `DIE_<n>_`-prefixed,
|
||
/// compared case-insensitively. Never watched. A probe run records it
|
||
/// beside the CPU rails because it is the one Energy Model rail that moves
|
||
/// on hosts whose CPU rails hold (the M5 Max and the M3 Ultra on macOS
|
||
/// 27.0.1), so a sampler that stopped sampling shows there.
|
||
pub(crate) fn is_gpu_energy_rail(name: &str) -> bool {
|
||
strip_die_prefix(name).eq_ignore_ascii_case("GPU Energy")
|
||
}
|
||
|
||
/// One CPU rail's identity within a sample: its channel name and its
|
||
/// ordinal among the channels of that name in the same sample. An M3
|
||
/// Ultra's Energy Model group lists 264 of its 645 channel names twice
|
||
/// (once per die, with no `DIE_` prefix); the CPU aggregates are uniquely
|
||
/// named today, but two same-named rails stepping together in one refresh
|
||
/// must count as one advance each, never as two advances of one rail.
|
||
pub(crate) type RailKey = (String, u32);
|
||
|
||
/// The channels of one sample whose names pass `keep`, keyed by
|
||
/// [`RailKey`] (the ordinal counts same-named channels among those kept),
|
||
/// in sample order.
|
||
fn keyed(sample: &[(String, i64)], keep: impl Fn(&str) -> bool) -> Vec<(RailKey, i64)> {
|
||
let mut rails: Vec<(RailKey, i64)> = Vec::new();
|
||
for (name, raw) in sample {
|
||
if !keep(name) {
|
||
continue;
|
||
}
|
||
let ordinal = rails.iter().filter(|((n, _), _)| n == name).count() as u32;
|
||
rails.push(((name.clone(), ordinal), *raw));
|
||
}
|
||
rails
|
||
}
|
||
|
||
/// The liveness rule over a sequence of samples.
|
||
///
|
||
/// Each usable sample is compared with the PREVIOUS usable one — never
|
||
/// with the first — on each rail's absolute raw integer, before unit
|
||
/// conversion (a small advance in a fine unit cannot truncate to zero). A
|
||
/// rail advances in an interval when its value rose over that interval; so
|
||
/// a rail advances at most once per interval, and each rail is counted on
|
||
/// its own: two dies stepping twice each never add up to a live rail. The
|
||
/// summed total is never consulted, because the GPU rail, which is live on
|
||
/// an M5 Max, is exactly what corrupts it. After each interval every rail
|
||
/// is put to [`sustained`], over the intervals observed so far and the wall
|
||
/// time since the first usable sample; the first time EVERY rail seen in
|
||
/// an interval passes, the watch says Live, and stays Live.
|
||
///
|
||
/// Every rail, not any one: the counter a Live verdict admits reads the
|
||
/// sum of every rail the watch picks. On a two-die part (an M3 Ultra's
|
||
/// `DIE_0_CPU Energy` and `DIE_1_CPU Energy`), one die live and the other
|
||
/// frozen would give a figure that leaves out the frozen die's energy,
|
||
/// or carries its refreshes, while claiming the whole CPU. A rail that
|
||
/// joins late is held to the intervals observed since the first, so it
|
||
/// is live only once its own advances are 80% of all of them. RAPL's
|
||
/// figure sums every package counter, so on a multi-socket server each
|
||
/// package is a rail, and every one must be live; NVML's one channel is
|
||
/// every rail it has.
|
||
///
|
||
/// Which channels of a sample are rails is the watch's `pick`: the CPU
|
||
/// energy aggregates for IOReport ([`RailWatch::default`]), every channel
|
||
/// for a single vendor counter ([`RailWatch::every_channel`]).
|
||
#[derive(Debug)]
|
||
pub(crate) struct RailWatch {
|
||
/// Which channels of a sample are watched as rails.
|
||
pick: fn(&str) -> bool,
|
||
/// The rails of the last usable sample.
|
||
prev: Option<Vec<(RailKey, i64)>>,
|
||
/// The wall-clock reading of the first usable sample.
|
||
first: Option<Duration>,
|
||
/// Intervals observed: pairs of consecutive usable samples.
|
||
pub(crate) intervals: u32,
|
||
/// Wall time from the first usable sample to the last.
|
||
pub(crate) window: Duration,
|
||
/// Every CPU rail seen in an interval, with the number of intervals in
|
||
/// which it advanced.
|
||
pub(crate) rails: Vec<(RailKey, u32)>,
|
||
/// Whether every rail has passed [`sustained`] after the same interval.
|
||
live: bool,
|
||
}
|
||
|
||
impl Default for RailWatch {
|
||
/// A watch of IOReport's CPU energy rails ([`is_cpu_energy_rail`]).
|
||
fn default() -> Self {
|
||
Self::of(is_cpu_energy_rail)
|
||
}
|
||
}
|
||
|
||
impl RailWatch {
|
||
/// A watch of the channels `pick` accepts.
|
||
fn of(pick: fn(&str) -> bool) -> Self {
|
||
Self { pick, prev: None, first: None, intervals: 0, window: Duration::ZERO, rails: Vec::new(), live: false }
|
||
}
|
||
|
||
/// A watch of every channel of a sample: for a vendor counter (RAPL's
|
||
/// package counters, one channel a package; NVML's total-energy
|
||
/// counter), whose every channel counts the domain it claims.
|
||
#[cfg_attr(not(any(target_os = "linux", target_os = "macos", feature = "cuda")), allow(dead_code))] // used by RAPL, NVML and the process counter
|
||
pub(crate) fn every_channel() -> Self {
|
||
Self::of(|_| true)
|
||
}
|
||
|
||
/// Fold in one usable sample: every channel in it, as (name, absolute
|
||
/// raw integer), and `at`, the wall-clock reading when the sample was
|
||
/// in hand. A sample that could not be taken is simply not observed,
|
||
/// so the next usable one pairs with the last usable one.
|
||
pub(crate) fn observe(&mut self, sample: &[(String, i64)], at: Duration) {
|
||
let cur = keyed(sample, self.pick);
|
||
let first = *self.first.get_or_insert(at);
|
||
self.window = at.saturating_sub(first);
|
||
if let Some(prev) = &self.prev {
|
||
self.intervals += 1;
|
||
for (key, now) in &cur {
|
||
let advanced = prev
|
||
.iter()
|
||
.find(|(k, _)| k == key)
|
||
.is_some_and(|(_, before)| now > before);
|
||
match self.rails.iter_mut().find(|(k, _)| k == key) {
|
||
Some((_, n)) => *n += u32::from(advanced),
|
||
None => self.rails.push((key.clone(), u32::from(advanced))),
|
||
}
|
||
}
|
||
let (intervals, window) = (self.intervals, self.window);
|
||
// Every rail, and at least one: `all` of no rails is vacuously
|
||
// true, and a sample with no rail in it is Absent, never Live.
|
||
self.live |= !self.rails.is_empty() && self.rails.iter().all(|(_, n)| sustained(*n, intervals, window));
|
||
}
|
||
self.prev = Some(cur);
|
||
}
|
||
|
||
/// Has every rail kept advancing, each by [`sustained`], after some
|
||
/// interval observed so far?
|
||
pub(crate) fn is_live(&self) -> bool {
|
||
self.live
|
||
}
|
||
|
||
/// Keep sampling? Until the rails prove live; or until `budget` has
|
||
/// elapsed **and** at least [`MIN_LIVE_ADVANCES`] intervals were
|
||
/// observed, enough for a live rail to have shown its advances (slow or
|
||
/// failed samples must not make a live rail look frozen); and never
|
||
/// past [`PROBE_BUDGET_CAP`] budgets.
|
||
pub(crate) fn wants_more(&self, elapsed: Duration, budget: Duration) -> bool {
|
||
!self.is_live()
|
||
&& elapsed < budget * PROBE_BUDGET_CAP
|
||
&& (elapsed < budget || self.intervals < MIN_LIVE_ADVANCES)
|
||
}
|
||
|
||
/// The verdict, or `None` when not one interval could be observed.
|
||
pub(crate) fn verdict(&self) -> Option<CpuRail> {
|
||
if self.intervals == 0 {
|
||
None
|
||
} else if self.is_live() {
|
||
Some(CpuRail::Live)
|
||
} else if self.rails.is_empty() {
|
||
Some(CpuRail::Absent)
|
||
} else {
|
||
Some(CpuRail::Frozen)
|
||
}
|
||
}
|
||
}
|
||
|
||
/// The probe loop, generic over how a sample is taken, what happens between
|
||
/// samples, and the clock, so it runs under test exactly as it runs on
|
||
/// hardware. Takes a sample, then ticks and samples again for as long as
|
||
/// [`RailWatch::wants_more`] says. `sample` returns `None` when no sample
|
||
/// could be taken; each one taken is handed to the rule with the wall
|
||
/// clock `now` read as it returns. [`probe_with`] runs it with the
|
||
/// sampler, tick and wall clock it is given, and records each sample call.
|
||
pub(crate) fn watch(
|
||
sample: impl FnMut() -> Option<Vec<(String, i64)>>,
|
||
tick: impl FnMut(),
|
||
now: impl Fn() -> Duration,
|
||
budget: Duration,
|
||
) -> RailWatch {
|
||
watch_from(RailWatch::default(), sample, tick, now, budget)
|
||
}
|
||
|
||
/// [`watch`], folding samples into the watch `w` it is given (so a
|
||
/// single vendor counter's probe watches every channel).
|
||
pub(crate) fn watch_from(
|
||
mut w: RailWatch,
|
||
mut sample: impl FnMut() -> Option<Vec<(String, i64)>>,
|
||
mut tick: impl FnMut(),
|
||
now: impl Fn() -> Duration,
|
||
budget: Duration,
|
||
) -> RailWatch {
|
||
let started = now();
|
||
if let Some(s) = sample() {
|
||
w.observe(&s, now());
|
||
}
|
||
while w.wants_more(now().saturating_sub(started), budget) {
|
||
tick();
|
||
if let Some(s) = sample() {
|
||
w.observe(&s, now());
|
||
}
|
||
}
|
||
w
|
||
}
|
||
|
||
/// Spin on the calling thread until `thread_cpu`, this thread's CPU clock,
|
||
/// has advanced by `want`, or `wall` has advanced by `wall_cap`, whichever
|
||
/// comes first. Returns the thread CPU consumed.
|
||
///
|
||
/// The target is CPU time, not wall time, so every tick burns the same CPU
|
||
/// however loaded the host is: while the thread is descheduled its CPU
|
||
/// clock stands still, and the tick simply takes longer. Only the wall cap
|
||
/// can cut a tick short, on a host that gives this thread less than
|
||
/// `want / wall_cap` of a core (5% for the probe's 25 ms in 500 ms).
|
||
pub(crate) fn spin_for_cpu(
|
||
thread_cpu: impl Fn() -> Duration,
|
||
wall: impl Fn() -> Duration,
|
||
want: Duration,
|
||
wall_cap: Duration,
|
||
) -> Duration {
|
||
let (cpu0, wall0) = (thread_cpu(), wall());
|
||
let mut acc: u64 = 0;
|
||
loop {
|
||
let used = thread_cpu().saturating_sub(cpu0);
|
||
if used >= want || wall().saturating_sub(wall0) >= wall_cap {
|
||
std::hint::black_box(acc);
|
||
return used;
|
||
}
|
||
for _ in 0..256 {
|
||
acc = std::hint::black_box(acc.wrapping_add(1));
|
||
}
|
||
}
|
||
}
|
||
|
||
/// One sample, as a sampler returns it: the shape of the IOReport sample it
|
||
/// read, and that sample's Energy Model channels.
|
||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||
pub(crate) struct Sample {
|
||
/// Channels in the IOReport sample, in any group.
|
||
pub(crate) channels: usize,
|
||
/// Whether every one of those channels is in the Energy Model group.
|
||
pub(crate) group_only: bool,
|
||
/// Each Energy Model channel's (name, absolute raw integer), in sample
|
||
/// order: what the probe hands the rule.
|
||
pub(crate) values: Vec<(String, i64)>,
|
||
}
|
||
|
||
/// One call the probe made to its sampler, as the probe itself recorded it,
|
||
/// so a test can pin what the probe did without timing it from outside.
|
||
#[derive(Debug, Clone)]
|
||
#[cfg_attr(not(test), allow(dead_code))] // read only by the tests
|
||
pub(crate) struct SampleCall {
|
||
/// Wall time of the call. Recorded, never asserted: on a shared host
|
||
/// another process's all-channel sample can hold IOReport for hundreds
|
||
/// of milliseconds, and this call waits for it.
|
||
pub(crate) wall: Duration,
|
||
/// The probing thread's CPU clock just before and just after the call.
|
||
pub(crate) cpu_before: Duration,
|
||
pub(crate) cpu_after: Duration,
|
||
/// For a sample that was taken: how many channels it held, and whether
|
||
/// every one is in the Energy Model group. `None` for no sample.
|
||
pub(crate) shape: Option<(usize, bool)>,
|
||
/// For a sample that was taken: every CPU rail and GPU Energy aggregate
|
||
/// in what was handed to the rule, as ([`RailKey`], raw), in sample
|
||
/// order. `None` for no sample.
|
||
pub(crate) handed: Option<Vec<(RailKey, i64)>>,
|
||
}
|
||
|
||
/// What one liveness probe did and saw.
|
||
#[derive(Debug)]
|
||
#[cfg_attr(not(test), allow(dead_code))] // all but `watch` read only by the tests
|
||
pub(crate) struct ProbeRun {
|
||
/// Whether the opener returned a sampler.
|
||
pub(crate) opened: bool,
|
||
/// Ticks run between samples.
|
||
pub(crate) ticks: u32,
|
||
/// Each tick's wall time, in order. Recorded, never decided on: a test
|
||
/// reads it to tell a tick that its wall cap cut short, on a host that
|
||
/// gave the thread almost no CPU, from one that did not burn at all.
|
||
pub(crate) tick_walls: Vec<Duration>,
|
||
/// Every sample call, in order.
|
||
pub(crate) calls: Vec<SampleCall>,
|
||
/// The rule, folded over every sample that was taken.
|
||
pub(crate) watch: RailWatch,
|
||
}
|
||
|
||
impl ProbeRun {
|
||
/// The rule's verdict: `None` when not one interval was observed, and
|
||
/// so also when the opener returned no sampler.
|
||
pub(crate) fn verdict(&self) -> Option<CpuRail> {
|
||
self.watch.verdict()
|
||
}
|
||
}
|
||
|
||
/// The liveness probe, generic over its real-world inputs so that tests
|
||
/// drive the very composition the production status runs: `open` makes the
|
||
/// sampler (`None`: no subscription, so no sample and no tick), `tick` runs
|
||
/// between samples, and `thread_cpu` and `wall` are the clocks. Samples by
|
||
/// the rule in [`watch`] for `budget`, and records every sample call.
|
||
pub(crate) fn probe_with<S>(
|
||
open: impl FnOnce() -> Option<S>,
|
||
mut tick: impl FnMut(),
|
||
thread_cpu: impl Fn() -> Duration,
|
||
wall: impl Fn() -> Duration,
|
||
budget: Duration,
|
||
) -> ProbeRun
|
||
where
|
||
S: FnMut() -> Option<Sample>,
|
||
{
|
||
let Some(mut sampler) = open() else {
|
||
return ProbeRun { opened: false, ticks: 0, tick_walls: Vec::new(), calls: Vec::new(), watch: RailWatch::default() };
|
||
};
|
||
let (mut calls, mut ticks, mut tick_walls) = (Vec::new(), 0u32, Vec::new());
|
||
let watch = watch(
|
||
|| {
|
||
let (cpu_before, started) = (thread_cpu(), wall());
|
||
let sample = sampler();
|
||
let (ended, cpu_after) = (wall(), thread_cpu());
|
||
calls.push(SampleCall {
|
||
wall: ended.saturating_sub(started),
|
||
cpu_before,
|
||
cpu_after,
|
||
shape: sample.as_ref().map(|s| (s.channels, s.group_only)),
|
||
handed: sample.as_ref().map(|s| keyed(&s.values, |n| is_cpu_energy_rail(n) || is_gpu_energy_rail(n))),
|
||
});
|
||
sample.map(|s| s.values)
|
||
},
|
||
|| {
|
||
ticks += 1;
|
||
let at = wall();
|
||
tick();
|
||
tick_walls.push(wall().saturating_sub(at));
|
||
},
|
||
&wall,
|
||
budget,
|
||
);
|
||
ProbeRun { opened: true, ticks, tick_walls, calls, watch }
|
||
}
|
||
|
||
/// The status a probe verdict gives this host. No verdict — no
|
||
/// subscription, or not one interval observed — is `Unavailable`: nothing
|
||
/// was learned about the rail, and only `Live` makes the counter available.
|
||
/// Never `CpuRailAbsent`, which says samples were taken and no CPU rail was
|
||
/// in them.
|
||
pub(crate) fn status_of(verdict: Option<CpuRail>) -> crate::IoReportStatus {
|
||
match verdict {
|
||
None => crate::IoReportStatus::Unavailable,
|
||
Some(CpuRail::Absent) => crate::IoReportStatus::CpuRailAbsent,
|
||
Some(CpuRail::Frozen) => crate::IoReportStatus::CpuRailFrozen,
|
||
Some(CpuRail::Live) => crate::IoReportStatus::Live,
|
||
}
|
||
}
|
||
|
||
/// Whether a host with this status may use the IOReport counter: only when
|
||
/// its CPU rail is live. Absent, frozen or unavailable all decline, so
|
||
/// callers estimate rather than seal a reading that did not track the work.
|
||
/// The one gate every constructor and selection path passes through.
|
||
pub(crate) fn admits_counter(status: crate::IoReportStatus) -> bool {
|
||
status == crate::IoReportStatus::Live
|
||
}
|
||
|
||
/// One liveness probe of a single vendor energy counter (RAPL, NVML):
|
||
/// `open` it; if it opens, sample it — each sample its channels as
|
||
/// (name, absolute raw integer), every one of them a rail — `tick`
|
||
/// between samples, by the wall clock `now`, for `budget`, by the same
|
||
/// rule and stopping rule as the IOReport probe ([`watch`]). `None` when
|
||
/// it does not open: then nothing is sampled and nothing ticks. Generic
|
||
/// over each part so the tests drive this very function with a scripted
|
||
/// counter, tick and clock; each rung's cached probe passes the real ones.
|
||
#[cfg_attr(not(any(target_os = "linux", target_os = "macos", feature = "cuda")), allow(dead_code))] // used by RAPL, NVML and the process counter
|
||
pub(crate) fn probe_counter<C>(
|
||
open: impl FnOnce() -> Option<C>,
|
||
mut sample: impl FnMut(&C) -> Option<Vec<(String, i64)>>,
|
||
tick: impl FnMut(),
|
||
now: impl Fn() -> Duration,
|
||
budget: Duration,
|
||
) -> Option<RailWatch> {
|
||
let counter = open()?;
|
||
Some(watch_from(RailWatch::every_channel(), || sample(&counter), tick, now, budget))
|
||
}
|
||
|
||
/// The status a counter probe gives its rung. No probe (the counter did
|
||
/// not open) and not one interval observed (no two usable samples in five
|
||
/// budgets) are both `Unavailable`; only a rail that kept advancing is
|
||
/// `Live`.
|
||
#[cfg_attr(not(any(target_os = "linux", target_os = "macos", feature = "cuda")), allow(dead_code))] // used by RAPL, NVML and the process counter
|
||
pub(crate) fn counter_status(probed: Option<&RailWatch>) -> crate::CounterStatus {
|
||
match probed {
|
||
None => crate::CounterStatus::Unavailable,
|
||
Some(w) if w.intervals == 0 => crate::CounterStatus::Unavailable,
|
||
Some(w) if w.is_live() => crate::CounterStatus::Live,
|
||
Some(_) => crate::CounterStatus::Frozen,
|
||
}
|
||
}
|
||
|
||
/// Whether a counter rung with this status may be selected: only when it
|
||
/// is live. The one gate the RAPL and NVML constructors pass through.
|
||
#[cfg_attr(not(any(target_os = "linux", target_os = "macos", feature = "cuda")), allow(dead_code))] // used by RAPL, NVML and the process counter
|
||
pub(crate) fn admits_vendor_counter(status: crate::CounterStatus) -> bool {
|
||
status == crate::CounterStatus::Live
|
||
}
|
||
|
||
#[cfg(test)]
|
||
mod tests {
|
||
use super::*;
|
||
use std::cell::Cell;
|
||
|
||
const TICK: Duration = Duration::from_millis(25);
|
||
const BUDGET: Duration = Duration::from_millis(1000);
|
||
|
||
/// One probe interval on an unloaded M5 Max: a 25 ms tick and a ~6 ms
|
||
/// Energy Model sample.
|
||
const INTERVAL: Duration = Duration::from_millis(31);
|
||
|
||
/// The observed two-step refresh of the M5 Max's `CPU Energy`, as a
|
||
/// probe sampling every interval sees it when the steps land in
|
||
/// intervals `first` and `second` (`first` < `second`): held, stepped
|
||
/// +61,764,485, stepped +3,435, held.
|
||
fn two_step_refresh(i: usize, first: usize, second: usize) -> i64 {
|
||
let a = 2_643_843_378;
|
||
if i < first {
|
||
a
|
||
} else if i < second {
|
||
a + 61_764_485
|
||
} else {
|
||
a + 61_764_485 + 3_435
|
||
}
|
||
}
|
||
|
||
/// Fold samples `0..n` straight into a [`RailWatch`], the i-th stamped
|
||
/// `i * step` on the wall clock. Returns the index of the sample after
|
||
/// which the watch first said Live, if any, and the watch.
|
||
fn first_live(step: Duration, n: usize, generate: impl Fn(usize) -> Vec<(&'static str, i64)>) -> (Option<usize>, RailWatch) {
|
||
let mut w = RailWatch::default();
|
||
let mut live_at = None;
|
||
for i in 0..n {
|
||
let s: Vec<(String, i64)> = generate(i).into_iter().map(|(n, raw)| (n.to_string(), raw)).collect();
|
||
w.observe(&s, step * i as u32);
|
||
if w.is_live() && live_at.is_none() {
|
||
live_at = Some(i);
|
||
}
|
||
}
|
||
(live_at, w)
|
||
}
|
||
|
||
/// Run [`watch`] over scripted samples — `gen(i)` is the i-th sample,
|
||
/// `None` for one that could not be taken — with a clock that only the
|
||
/// tick moves, by `step`. Returns the watch and how many samples it took.
|
||
fn scripted(step: Duration, mut generate: impl FnMut(usize) -> Option<Vec<(&'static str, i64)>>) -> (RailWatch, usize) {
|
||
let clock = Cell::new(Duration::ZERO);
|
||
let mut taken = 0usize;
|
||
let w = watch(
|
||
|| {
|
||
assert!(taken < 100_000, "the probe never stopped sampling");
|
||
let s = generate(taken);
|
||
taken += 1;
|
||
s.map(|v| v.into_iter().map(|(n, raw)| (n.to_string(), raw)).collect())
|
||
},
|
||
|| clock.set(clock.get() + step),
|
||
|| clock.get(),
|
||
BUDGET,
|
||
);
|
||
(w, taken)
|
||
}
|
||
|
||
fn key(name: &str, ordinal: u32) -> RailKey {
|
||
(name.to_string(), ordinal)
|
||
}
|
||
|
||
#[test]
|
||
fn only_the_cpu_energy_aggregate_is_a_cpu_rail() {
|
||
for name in ["CPU Energy", "cpu energy", "DIE_0_CPU Energy", "DIE_1_CPU Energy", "DIE_12_CPU Energy"] {
|
||
assert!(is_cpu_energy_rail(name), "{name}");
|
||
}
|
||
for name in ["GPU Energy", "DIE_0_GPU Energy", "ANE Energy", "PCPU", "MCPU0", "PACC_0", "DIE_0_PACC0_CPU0", "ECPUDTL00", "CPU Energy X", "DIE__CPU"] {
|
||
assert!(!is_cpu_energy_rail(name), "{name}");
|
||
}
|
||
}
|
||
|
||
/// (a) The M5 Max refresh: `CPU Energy` held at 480,083,160, stepped
|
||
/// once to 541,893,638 and held, while `GPU Energy` moved at every
|
||
/// sample. One advance in 40 intervals: Frozen. A rule that compared
|
||
/// every sample with the first would see an advance in every interval
|
||
/// after the step and call it live.
|
||
#[test]
|
||
fn a_the_m5_max_refresh_is_frozen() {
|
||
let (w, taken) = scripted(TICK, |i| {
|
||
Some(vec![
|
||
("CPU Energy", if i < 2 { 480_083_160 } else { 541_893_638 }),
|
||
("GPU Energy", 1_000 + 7 * i as i64),
|
||
])
|
||
});
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "one refresh counted as a live rail: {w:?}");
|
||
assert_eq!(w.rails, vec![(key("CPU Energy", 0), 1)], "the refresh is one advance of one rail");
|
||
// The budget is 40 ticks; with five intervals long since seen, it stops there.
|
||
assert_eq!((w.intervals, taken), (40, 41));
|
||
}
|
||
|
||
/// (b) The M3 Ultra refresh: both dies step once, together, in one
|
||
/// sample, amid unprefixed per-die channels that repeat a name.
|
||
#[test]
|
||
fn b_the_m3_ultra_two_die_refresh_in_one_sample_is_frozen() {
|
||
let (w, _) = scripted(TICK, |i| {
|
||
let stepped = i >= 5;
|
||
Some(vec![
|
||
("DIE_0_CPU Energy", 9_494_336 + if stepped { 1_344_869 } else { 0 }),
|
||
("ECPUDTL00", 10 * i as i64),
|
||
("DIE_1_CPU Energy", 10_212_656 + if stepped { 1_391_637 } else { 0 }),
|
||
("ECPUDTL00", 11 * i as i64),
|
||
("GPU Energy", 3 * i as i64),
|
||
])
|
||
});
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "one refresh of both dies counted as live: {w:?}");
|
||
assert_eq!(w.rails, vec![(key("DIE_0_CPU Energy", 0), 1), (key("DIE_1_CPU Energy", 0), 1)]);
|
||
}
|
||
|
||
/// (c) Two channels with the same name are two rails: stepping together
|
||
/// once is one advance each — in the same interval or in different ones —
|
||
/// never two advances of one rail. Both are summed by the counter, so
|
||
/// both must prove live: one live beside one that holds is Frozen.
|
||
#[test]
|
||
fn c_two_same_named_cpu_rails_are_counted_apart() {
|
||
let (w, _) = scripted(TICK, |i| {
|
||
let j = if i >= 3 { 1_300_000 } else { 0 };
|
||
Some(vec![("CPU Energy", 100 + j), ("CPU Energy", 200 + j)])
|
||
});
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "one refresh of two same-named rails counted as live: {w:?}");
|
||
assert_eq!(w.rails, vec![(key("CPU Energy", 0), 1), (key("CPU Energy", 1), 1)]);
|
||
|
||
let (w, _) = scripted(TICK, |i| {
|
||
Some(vec![("CPU Energy", 100 + if i >= 3 { 5 } else { 0 }), ("CPU Energy", 200 + if i >= 7 { 5 } else { 0 })])
|
||
});
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "two same-named rails stepping once each counted as live: {w:?}");
|
||
|
||
// The first of two same-named rails is live; the second holds:
|
||
// Frozen, watched for the whole budget.
|
||
let (w, taken) = scripted(TICK, |i| Some(vec![("CPU Energy", 100 + i as i64), ("CPU Energy", 200)]));
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "one live rail beside one that holds counted as live: {w:?}");
|
||
assert_eq!((w.intervals, taken), (40, 41));
|
||
assert_eq!(w.rails, vec![(key("CPU Energy", 0), 40), (key("CPU Energy", 1), 0)]);
|
||
|
||
// Both live. At 25 ms a sample, the 300 ms window is what they wait
|
||
// for: 12 intervals.
|
||
let (w, taken) = scripted(TICK, |i| Some(vec![("CPU Energy", 100 + i as i64), ("CPU Energy", 200 + 3 * i as i64)]));
|
||
assert_eq!(w.verdict(), Some(CpuRail::Live), "{w:?}");
|
||
assert_eq!((w.intervals, taken), (12, 13));
|
||
assert_eq!(w.rails, vec![(key("CPU Energy", 0), 12), (key("CPU Energy", 1), 12)]);
|
||
}
|
||
|
||
/// (d) A rail advancing in every interval, sampled every 31 ms, is not
|
||
/// Live at its fifth advance (155 ms), nor at its ninth (279 ms): only
|
||
/// at the tenth, once the window reaches 300 ms (310 ms). Sampling stops
|
||
/// right there. Five advances and the window are both needed, and the
|
||
/// window is inclusive: at 60 ms a sample, Live at the fifth interval
|
||
/// (exactly 300 ms); at 59 ms, not until the sixth (295 ms is short).
|
||
#[test]
|
||
fn d_a_rail_advancing_every_interval_is_live_at_five_advances_and_300_ms() {
|
||
let live = |i: usize| vec![("GPU Energy", 5), ("CPU Energy", 1_000 + 13 * i as i64)];
|
||
let (at, w) = first_live(INTERVAL, 40, live);
|
||
assert_eq!(at, Some(10), "a rail advancing every 31 ms must be Live at the 10th interval (310 ms), not before: {w:?}");
|
||
let (at, w) = first_live(INTERVAL, 10, live);
|
||
assert_eq!((at, w.intervals, w.window), (None, 9, Duration::from_millis(279)), "Live before 300 ms: {w:?}");
|
||
assert_eq!(w.rails, vec![(key("CPU Energy", 0), 9)]);
|
||
|
||
let (w, taken) = scripted(INTERVAL, |i| Some(live(i)));
|
||
assert_eq!(w.verdict(), Some(CpuRail::Live));
|
||
assert_eq!((w.intervals, taken), (10, 11), "a live rail must stop the probe as soon as the rule holds");
|
||
assert_eq!((w.rails.clone(), w.window), (vec![(key("CPU Energy", 0), 10)], Duration::from_millis(310)));
|
||
|
||
assert_eq!(first_live(Duration::from_millis(60), 20, live).0, Some(5), "300 ms is enough");
|
||
assert_eq!(first_live(Duration::from_millis(59), 20, live).0, Some(6), "295 ms is not");
|
||
}
|
||
|
||
/// The rule's three bars, at their edges: five advances, 80% of the
|
||
/// intervals observed, 300 ms of window.
|
||
#[test]
|
||
fn sustained_needs_five_advances_80_percent_and_300_ms() {
|
||
let ms = Duration::from_millis;
|
||
assert!(sustained(5, 5, ms(300)));
|
||
assert!(!sustained(4, 4, ms(1000)), "four advances");
|
||
assert!(!sustained(5, 5, ms(299)), "299 ms");
|
||
assert!(sustained(5, 6, ms(300)), "5 of 6 is 83%");
|
||
assert!(!sustained(5, 7, ms(1000)), "5 of 7 is 71%");
|
||
assert!(sustained(8, 10, ms(300)), "8 of 10 is 80%");
|
||
assert!(!sustained(7, 9, ms(1000)), "7 of 9 is 78%");
|
||
assert!(sustained(800, 1000, ms(5000)) && !sustained(799, 1000, ms(5000)));
|
||
assert!(!sustained(2, 2, ms(1000)), "two advances: the old rule's Live");
|
||
}
|
||
|
||
/// (h) The observed macOS 27 refresh published in two steps, +61,764,485
|
||
/// then +3,435 (64 ms apart, so in consecutive or next-but-one 31 ms
|
||
/// intervals), wherever it falls in the probe, beside a moving GPU rail
|
||
/// and sub-rails that step with it: two advances, Frozen. The old rule
|
||
/// (two advancing intervals) called every one of these Live.
|
||
#[test]
|
||
fn h_the_two_step_refresh_is_frozen() {
|
||
for (first, second) in [(3, 4), (1, 2), (1, 3), (3, 5), (20, 21), (31, 33)] {
|
||
let sample = |i: usize| {
|
||
let cpu = two_step_refresh(i, first, second);
|
||
vec![
|
||
("CPU Energy", cpu),
|
||
("MCPU0", cpu / 4),
|
||
("MCPU1", cpu / 4),
|
||
("PCPU", cpu / 2),
|
||
("GPU Energy", 1_000 + 7 * i as i64),
|
||
]
|
||
};
|
||
let (w, taken) = scripted(INTERVAL, |i| Some(sample(i)));
|
||
assert_eq!(
|
||
w.verdict(),
|
||
Some(CpuRail::Frozen),
|
||
"the two-step refresh in intervals {first} and {second} counted as a live rail: {w:?}"
|
||
);
|
||
assert_eq!(w.rails, vec![(key("CPU Energy", 0), 2)], "steps in intervals {first} and {second}");
|
||
assert_eq!((w.intervals, taken), (33, 34), "the budget is 33 intervals of 31 ms");
|
||
assert_eq!(first_live(INTERVAL, 200, sample).0, None, "steps in intervals {first} and {second}");
|
||
}
|
||
}
|
||
|
||
/// (i) A refresh published in three, four or five steps within 150 ms
|
||
/// at the very start of the probe (30 ms a sample): while it lasts it is
|
||
/// every interval observed, and at five steps it has five advances, but
|
||
/// the window is under 300 ms; by the time it is not, the rail has held
|
||
/// long enough that its advances are under 80% of the intervals. Frozen.
|
||
#[test]
|
||
fn i_a_refresh_in_three_to_five_steps_at_the_start_is_frozen() {
|
||
let step = Duration::from_millis(30);
|
||
let sizes = [61_764_485, 3_435, 1_207, 88, 19];
|
||
for steps in 3..=5usize {
|
||
let sample = |i: usize| vec![("CPU Energy", 2_643_843_378 + sizes[..i.min(steps)].iter().sum::<i64>())];
|
||
let (w, _) = scripted(step, |i| Some(sample(i)));
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "a refresh in {steps} steps counted as a live rail: {w:?}");
|
||
assert_eq!(w.rails, vec![(key("CPU Energy", 0), steps as u32)]);
|
||
assert_eq!(first_live(step, 200, sample).0, None, "a refresh in {steps} steps");
|
||
}
|
||
}
|
||
|
||
/// (j) The M3 Ultra's shape: `DIE_0_CPU Energy` and `DIE_1_CPU Energy`
|
||
/// each step twice within 64 ms (two consecutive 31 ms intervals),
|
||
/// together or one interval apart, amid unprefixed per-die channels that
|
||
/// repeat a name; at the start of the probe and inside it. Two advances
|
||
/// per die: Frozen. The old rule called each die Live.
|
||
#[test]
|
||
fn j_the_m3_ultra_two_steps_on_each_die_are_frozen() {
|
||
for (d0, d1) in [(1, 1), (1, 2), (6, 6), (6, 7)] {
|
||
let sample = |i: usize| {
|
||
let steps = |from: usize| i64::from(i >= from) * 1_344_869 + i64::from(i > from) * 2_113;
|
||
vec![
|
||
("DIE_0_CPU Energy", 9_494_336 + steps(d0)),
|
||
("ECPUDTL00", 10 * i as i64),
|
||
("DIE_1_CPU Energy", 10_212_656 + steps(d1)),
|
||
("ECPUDTL00", 11 * i as i64),
|
||
("GPU Energy", 3 * i as i64),
|
||
]
|
||
};
|
||
let (w, _) = scripted(INTERVAL, |i| Some(sample(i)));
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "two steps on each die ({d0}, {d1}) counted as live: {w:?}");
|
||
assert_eq!(w.rails, vec![(key("DIE_0_CPU Energy", 0), 2), (key("DIE_1_CPU Energy", 0), 2)]);
|
||
assert_eq!(first_live(INTERVAL, 200, sample).0, None, "dies stepping from {d0} and {d1}");
|
||
}
|
||
}
|
||
|
||
/// (k) A live rail that misses one interval in every five advances in
|
||
/// exactly 80% of them: Live, at the tenth interval (8 of 10, 310 ms).
|
||
/// One that missed two of its first five is not yet Live at the fifth
|
||
/// (3 of 5), nor at the ninth (7 of 9, 78%), though it advanced in every
|
||
/// interval since; it is at the tenth (8 of 10) — here sampled every
|
||
/// 100 ms, so the window never binds. And one that misses two in every
|
||
/// five (60%) is never Live: Frozen.
|
||
#[test]
|
||
fn k_a_live_rail_missing_one_interval_in_five_is_live_two_is_not_yet() {
|
||
let miss_one = |i: usize| vec![("CPU Energy", (1..=i).filter(|k| k % 5 != 0).count() as i64)];
|
||
let (at, w) = first_live(INTERVAL, 40, miss_one);
|
||
assert_eq!(at, Some(10), "missing one in five must be Live at 8 of 10 intervals: {w:?}");
|
||
let (w, taken) = scripted(INTERVAL, |i| Some(miss_one(i)));
|
||
assert_eq!((w.verdict(), w.intervals, taken), (Some(CpuRail::Live), 10, 11));
|
||
assert_eq!(w.rails, vec![(key("CPU Energy", 0), 8)]);
|
||
|
||
let miss_first_two = |i: usize| vec![("CPU Energy", (1..=i).filter(|k| *k != 1 && *k != 3).count() as i64)];
|
||
let step = Duration::from_millis(100);
|
||
for n in [6, 10] {
|
||
let (at, w) = first_live(step, n, miss_first_two);
|
||
assert_eq!(at, None, "Live at {} of {} intervals: {w:?}", w.rails[0].1, w.intervals);
|
||
}
|
||
let (at, w) = first_live(step, 40, miss_first_two);
|
||
assert_eq!(at, Some(10), "8 of 10 intervals must be Live: {w:?}");
|
||
|
||
let miss_two = |i: usize| vec![("CPU Energy", (1..=i).filter(|k| k % 5 != 0 && k % 5 != 2).count() as i64)];
|
||
let (w, _) = scripted(INTERVAL, |i| Some(miss_two(i)));
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "a rail advancing in 60% of intervals counted as live: {w:?}");
|
||
assert_eq!(first_live(INTERVAL, 1000, miss_two).0, None);
|
||
}
|
||
|
||
/// (l) A loaded host, where each interval takes 100 ms: a live rail is
|
||
/// Live after exactly five intervals (500 ms). By 300 ms it has only
|
||
/// three advances. Sampling stops right there.
|
||
#[test]
|
||
fn l_a_live_rail_on_a_loaded_host_is_live_after_five_intervals() {
|
||
let step = Duration::from_millis(100);
|
||
let live = |i: usize| vec![("CPU Energy", 1_000 + 13 * i as i64)];
|
||
let (at, w) = first_live(step, 40, live);
|
||
assert_eq!(at, Some(5), "{w:?}");
|
||
let (w, taken) = scripted(step, |i| Some(live(i)));
|
||
assert_eq!((w.verdict(), w.intervals, taken, w.window), (Some(CpuRail::Live), 5, 6, Duration::from_millis(500)));
|
||
}
|
||
|
||
/// (m) Every die must sustain its own advance, because the counter
|
||
/// sums every die's `CPU Energy`. On an M3 Ultra's two dies, sampled
|
||
/// every 31 ms beside a moving GPU rail and unprefixed per-die channels
|
||
/// that repeat a name:
|
||
/// - both dies advancing at every interval: Live, at the tenth interval
|
||
/// (310 ms), exactly where one die alone is (d);
|
||
/// - `DIE_0` advancing at every interval and `DIE_1` frozen — holding,
|
||
/// stepping once, or stepping twice in a refresh: Frozen, however
|
||
/// long it is watched. A rule that took any one live rail called each
|
||
/// of these Live at the tenth interval, and the counter it admitted
|
||
/// would leave `DIE_1` out of a figure claimed for the whole CPU;
|
||
/// - `DIE_0` live and `DIE_1` advancing in 60% of intervals: Frozen;
|
||
/// - `DIE_1` frozen for its first twenty intervals, then live: Live
|
||
/// only once its own advances reach 80% of every interval observed
|
||
/// (at the hundredth, 80 of 100), never at `DIE_0`'s tenth.
|
||
#[test]
|
||
fn m_every_die_must_sustain_its_own_advance() {
|
||
let dies = |die0: i64, die1: i64, i: usize| {
|
||
vec![
|
||
("DIE_0_CPU Energy", die0),
|
||
("ECPUDTL00", 10 * i as i64),
|
||
("DIE_1_CPU Energy", die1),
|
||
("ECPUDTL00", 11 * i as i64),
|
||
("GPU Energy", 3 * i as i64),
|
||
]
|
||
};
|
||
let live0 = |i: usize| 9_494_336 + 1_307 * i as i64;
|
||
|
||
let both = |i: usize| dies(live0(i), 10_212_656 + 1_211 * i as i64, i);
|
||
let (at, w) = first_live(INTERVAL, 40, both);
|
||
assert_eq!(at, Some(10), "two live dies must be Live at the 10th interval (310 ms), as one is: {w:?}");
|
||
let (w, taken) = scripted(INTERVAL, |i| Some(both(i)));
|
||
assert_eq!((w.verdict(), w.intervals, taken), (Some(CpuRail::Live), 10, 11));
|
||
assert_eq!(w.rails, vec![(key("DIE_0_CPU Energy", 0), 10), (key("DIE_1_CPU Energy", 0), 10)]);
|
||
|
||
// DIE_1 frozen: what it does, its value at sample i, its advances.
|
||
let frozen = [
|
||
("holds", (|_| 10_212_656) as fn(usize) -> i64, 0),
|
||
("steps once", |i| 10_212_656 + i64::from(i >= 4) * 1_391_637, 1),
|
||
("steps twice in a refresh", |i| 10_212_656 + i64::from(i >= 4) * 1_391_637 + i64::from(i >= 5) * 2_113, 2),
|
||
];
|
||
for (what, die1, steps) in frozen {
|
||
let mixed = |i: usize| dies(live0(i), die1(i), i);
|
||
let (w, taken) = scripted(INTERVAL, |i| Some(mixed(i)));
|
||
assert_eq!(
|
||
w.verdict(),
|
||
Some(CpuRail::Frozen),
|
||
"DIE_0 live and DIE_1 frozen ({what}) counted as live: the counter would leave DIE_1 out: {w:?}"
|
||
);
|
||
assert_eq!(w.rails, vec![(key("DIE_0_CPU Energy", 0), 33), (key("DIE_1_CPU Energy", 0), steps)], "{what}");
|
||
assert_eq!((w.intervals, taken), (33, 34), "{what}: the budget is 33 intervals of 31 ms");
|
||
assert_eq!(first_live(INTERVAL, 1000, mixed).0, None, "DIE_1 {what}");
|
||
}
|
||
|
||
let sixty = |i: usize| dies(live0(i), 10_212_656 + (1..=i).filter(|k| k % 5 != 0 && k % 5 != 2).count() as i64, i);
|
||
let (w, _) = scripted(INTERVAL, |i| Some(sixty(i)));
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "DIE_1 advancing in 60% of intervals counted as live: {w:?}");
|
||
assert_eq!(first_live(INTERVAL, 1000, sixty).0, None);
|
||
|
||
let late = |i: usize| dies(live0(i), 10_212_656 + i.saturating_sub(20) as i64, i);
|
||
let (at, w) = first_live(INTERVAL, 200, late);
|
||
assert_eq!(at, Some(100), "DIE_1 live from the 21st interval must be Live at 80 of 100: {w:?}");
|
||
assert_eq!(w.rails, vec![(key("DIE_0_CPU Energy", 0), 199), (key("DIE_1_CPU Energy", 0), 179)]);
|
||
let (w, _) = scripted(INTERVAL, |i| Some(late(i)));
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "DIE_1 live for only 13 of 33 intervals counted as live: {w:?}");
|
||
assert_eq!(w.rails, vec![(key("DIE_0_CPU Energy", 0), 33), (key("DIE_1_CPU Energy", 0), 13)]);
|
||
}
|
||
|
||
/// (e) Samples that cannot be taken keep the probe sampling to its cap of
|
||
/// five budgets, and end with no verdict: Unavailable.
|
||
#[test]
|
||
fn e_failed_samples_sample_to_the_cap_and_give_no_verdict() {
|
||
let (w, taken) = scripted(TICK, |_| None);
|
||
assert_eq!(w.verdict(), None);
|
||
assert_eq!(status_of(w.verdict()), crate::IoReportStatus::Unavailable);
|
||
assert_eq!(taken, 5 * 40 + 1, "the probe must keep sampling to five budgets");
|
||
// One usable sample is still no interval.
|
||
let (w, taken) = scripted(TICK, |i| (i == 0).then(|| vec![("CPU Energy", 7)]));
|
||
assert_eq!((w.verdict(), w.intervals, taken), (None, 0, 201));
|
||
}
|
||
|
||
/// (f) Past the budget, the probe stops at once if it has seen five
|
||
/// intervals; with fewer it keeps sampling until it has five, or hits
|
||
/// the cap of five budgets.
|
||
#[test]
|
||
fn f_the_budget_stops_the_probe_only_once_five_intervals_are_seen() {
|
||
// Every tick takes 600 ms: past the budget after two, it still
|
||
// takes six samples, and stops at the fifth interval (3 s).
|
||
let (w, taken) = scripted(Duration::from_millis(600), |_| Some(vec![("CPU Energy", 42)]));
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen));
|
||
assert_eq!((w.intervals, taken, w.window), (5, 6, Duration::from_millis(3000)));
|
||
// A live rail sampled that slowly is Live at its fifth advance.
|
||
let (w, taken) = scripted(Duration::from_millis(600), |i| Some(vec![("CPU Energy", i as i64)]));
|
||
assert_eq!((w.verdict(), w.intervals, taken), (Some(CpuRail::Live), 5, 6));
|
||
// One interval, then failures: past the budget it keeps sampling, to the cap.
|
||
let (w, taken) = scripted(TICK, |i| (i < 2).then(|| vec![("CPU Energy", 42)]));
|
||
assert_eq!((w.verdict(), w.intervals, taken), (Some(CpuRail::Frozen), 1, 201));
|
||
// Four intervals, then failures: still to the cap.
|
||
let (w, taken) = scripted(TICK, |i| (i < 5).then(|| vec![("CPU Energy", i as i64)]));
|
||
assert_eq!((w.verdict(), w.intervals, taken), (Some(CpuRail::Frozen), 4, 201));
|
||
// The fifth interval arrives at 2.5 s, past the budget: it stops right there.
|
||
let (w, taken) = scripted(TICK, |i| (i < 5 || i == 100).then(|| vec![("CPU Energy", 42)]));
|
||
assert_eq!((w.verdict(), w.intervals, taken), (Some(CpuRail::Frozen), 5, 101));
|
||
// Within the budget, five intervals are not enough to stop.
|
||
let (w, taken) = scripted(Duration::from_millis(100), |_| Some(vec![("CPU Energy", 42)]));
|
||
assert_eq!((w.intervals, taken), (10, 11));
|
||
// A tick of 2 s: the cap (5 s) stops it at three intervals, short of
|
||
// five, and a rail advancing in all three is still not Live.
|
||
let (w, taken) = scripted(Duration::from_secs(2), |i| Some(vec![("CPU Energy", i as i64)]));
|
||
assert_eq!((w.verdict(), w.intervals, taken), (Some(CpuRail::Frozen), 3, 4));
|
||
}
|
||
|
||
/// (g) DIE_0 steps once and DIE_1 steps once, in different intervals:
|
||
/// advances are counted per rail, never summed across dies.
|
||
#[test]
|
||
fn g_one_step_on_each_die_in_different_intervals_is_frozen() {
|
||
let (w, _) = scripted(TICK, |i| {
|
||
Some(vec![
|
||
("DIE_0_CPU Energy", if i >= 3 { 9_500_000 } else { 9_494_336 }),
|
||
("DIE_1_CPU Energy", if i >= 6 { 10_300_000 } else { 10_212_656 }),
|
||
])
|
||
});
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "one step per die counted as a live rail: {w:?}");
|
||
assert_eq!(w.rails, vec![(key("DIE_0_CPU Energy", 0), 1), (key("DIE_1_CPU Energy", 0), 1)]);
|
||
}
|
||
|
||
/// Samples with no CPU rail in them, however much else moves: Absent.
|
||
#[test]
|
||
fn no_cpu_rail_is_absent() {
|
||
let (w, _) = scripted(TICK, |i| Some(vec![("GPU Energy", i as i64), ("PCPU", i as i64)]));
|
||
assert_eq!(w.verdict(), Some(CpuRail::Absent));
|
||
assert!(w.rails.is_empty(), "{w:?}");
|
||
}
|
||
|
||
/// A rail that goes DOWN between consecutive samples — a reset or a
|
||
/// wrap — has not advanced. A rail falling at every sample is Frozen
|
||
/// with no advance at all; one real step followed by a reset is one
|
||
/// advance; and a rail that alternately rises and falls advances in
|
||
/// only half of its intervals, which is not the 80% a live rail needs.
|
||
#[test]
|
||
fn a_decrease_is_not_an_advance() {
|
||
let (w, _) = scripted(TICK, |i| Some(vec![("CPU Energy", 1_000 - i as i64)]));
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "a falling rail counted as live: {w:?}");
|
||
assert_eq!(w.rails, vec![(key("CPU Energy", 0), 0)], "a decrease counted as an advance");
|
||
|
||
let (w, _) = scripted(TICK, |i| Some(vec![("CPU Energy", [500, 500, 500, 900, 900, 900, 10][i.min(6)])]));
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "one step and one reset counted as live: {w:?}");
|
||
assert_eq!(w.rails, vec![(key("CPU Energy", 0), 1)], "the reset counted as an advance");
|
||
|
||
let (w, _) = scripted(INTERVAL, |i| Some(vec![("CPU Energy", 500 + (i % 2) as i64)]));
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "a rail rising and falling in turn counted as live: {w:?}");
|
||
assert_eq!(w.rails, vec![(key("CPU Energy", 0), 17)], "each fall counted as an advance");
|
||
}
|
||
|
||
/// A rail absent from the previous sample only sets its baseline in
|
||
/// this one; it has not advanced. A rail that joins late, at any value,
|
||
/// and holds is Frozen with no advance; and a rail that drops out of
|
||
/// every other sample is never "new, so advanced" when it comes back.
|
||
#[test]
|
||
fn a_rail_new_in_a_sample_only_sets_its_baseline() {
|
||
let (w, _) = scripted(TICK, |i| {
|
||
let mut s = vec![("DIE_0_CPU Energy", 100)];
|
||
if i >= 3 {
|
||
s.push(("DIE_1_CPU Energy", 5_000_000));
|
||
}
|
||
Some(s)
|
||
});
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "{w:?}");
|
||
assert_eq!(
|
||
w.rails,
|
||
vec![(key("DIE_0_CPU Energy", 0), 0), (key("DIE_1_CPU Energy", 0), 0)],
|
||
"a rail's first appearance counted as an advance"
|
||
);
|
||
|
||
let (w, _) = scripted(TICK, |i| {
|
||
Some(if i % 2 == 0 { vec![("CPU Energy", 7)] } else { vec![("GPU Energy", i as i64)] })
|
||
});
|
||
assert_eq!(w.verdict(), Some(CpuRail::Frozen), "a rail's reappearances counted as advances: {w:?}");
|
||
assert_eq!(w.rails, vec![(key("CPU Energy", 0), 0)]);
|
||
}
|
||
|
||
/// Every verdict maps to its status; only a live rail is `Live`.
|
||
#[test]
|
||
fn the_verdict_maps_to_the_status() {
|
||
use crate::IoReportStatus as S;
|
||
assert_eq!(status_of(None), S::Unavailable);
|
||
assert_eq!(status_of(Some(CpuRail::Absent)), S::CpuRailAbsent);
|
||
assert_eq!(status_of(Some(CpuRail::Frozen)), S::CpuRailFrozen);
|
||
assert_eq!(status_of(Some(CpuRail::Live)), S::Live);
|
||
}
|
||
|
||
/// No verdict (not one interval observed) is `Unavailable`: never
|
||
/// `Live`, never `CpuRailAbsent`.
|
||
#[test]
|
||
fn no_verdict_is_unavailable() {
|
||
let status = status_of(None);
|
||
assert_eq!(status, crate::IoReportStatus::Unavailable);
|
||
assert!(!admits_counter(status), "no verdict admitted the counter");
|
||
}
|
||
|
||
/// A sample the way the FFI returns one of the Energy Model group:
|
||
/// every channel in the group.
|
||
fn group_sample(values: &[(&str, i64)]) -> Sample {
|
||
Sample {
|
||
channels: values.len(),
|
||
group_only: true,
|
||
values: values.iter().map(|(n, raw)| (n.to_string(), *raw)).collect(),
|
||
}
|
||
}
|
||
|
||
/// What a fake run of [`probe_with`] did: the run, and how many times
|
||
/// the fake opener, sampler and tick were called.
|
||
struct Fake {
|
||
run: ProbeRun,
|
||
opens: u32,
|
||
samples: usize,
|
||
ticks: u32,
|
||
}
|
||
|
||
/// [`probe_with`] as the production status composes it (an opener, a
|
||
/// tick, a thread CPU clock, a wall clock, the budget), with fakes for
|
||
/// every real-world input: the opener opens or not; `generate(i)` is
|
||
/// the i-th sample; the wall clock moves only on a tick, by `step`, and
|
||
/// the thread CPU clock by 25 ms per tick and 1 ms per sample call.
|
||
fn fake(opens: bool, step: Duration, mut generate: impl FnMut(usize) -> Option<Sample>) -> Fake {
|
||
let (wall, cpu) = (Cell::new(Duration::ZERO), Cell::new(Duration::ZERO));
|
||
let (opened, ticked) = (Cell::new(0u32), Cell::new(0u32));
|
||
let mut taken = 0usize;
|
||
let sampler = || {
|
||
assert!(taken < 100_000, "the probe never stopped sampling");
|
||
let s = generate(taken);
|
||
taken += 1;
|
||
cpu.set(cpu.get() + Duration::from_millis(1));
|
||
s
|
||
};
|
||
let run = probe_with(
|
||
|| {
|
||
opened.set(opened.get() + 1);
|
||
opens.then_some(sampler)
|
||
},
|
||
|| {
|
||
ticked.set(ticked.get() + 1);
|
||
wall.set(wall.get() + step);
|
||
cpu.set(cpu.get() + TICK);
|
||
},
|
||
|| cpu.get(),
|
||
|| wall.get(),
|
||
BUDGET,
|
||
);
|
||
Fake { run, opens: opened.get(), samples: taken, ticks: ticked.get() }
|
||
}
|
||
|
||
/// No subscription: the opener is asked once, returns no sampler, and
|
||
/// the probe takes no sample and runs no tick. No verdict:
|
||
/// `Unavailable` — never `Live`, and never `CpuRailAbsent`. A
|
||
/// subscription that never yields two usable samples is `Unavailable`
|
||
/// too: nothing was learned about the CPU rail.
|
||
#[test]
|
||
fn no_energy_model_subscription_is_unavailable() {
|
||
use crate::IoReportStatus as S;
|
||
let f = fake(false, TICK, |_| panic!("sampled without a subscription"));
|
||
assert_eq!((f.opens, f.samples, f.ticks), (1, 0, 0), "no subscription: no sample, no tick");
|
||
assert!(!f.run.opened && f.run.calls.is_empty() && f.run.ticks == 0, "{:?}", f.run);
|
||
assert_eq!(f.run.verdict(), None);
|
||
assert_eq!(status_of(f.run.verdict()), S::Unavailable, "no group subscription");
|
||
let never = fake(true, TICK, |_| None);
|
||
assert_eq!(status_of(never.run.verdict()), S::Unavailable, "a subscription with no sample");
|
||
assert!(never.run.calls.iter().all(|c| c.shape.is_none() && c.handed.is_none()), "a failed sample recorded as taken");
|
||
let once = fake(true, TICK, |i| (i == 0).then(|| group_sample(&[("CPU Energy", 7)])));
|
||
assert_eq!(status_of(once.run.verdict()), S::Unavailable, "a subscription with one sample");
|
||
}
|
||
|
||
/// The M5 Max refresh through the whole composition: `CPU Energy`
|
||
/// steps once, `GPU Energy` moves at every sample. Frozen. Every tick
|
||
/// runs between two samples, and the run records, for every sample, its
|
||
/// shape, the thread CPU clock it was given, and exactly the CPU and GPU
|
||
/// rails it handed the rule.
|
||
#[test]
|
||
fn probe_with_a_single_batch_jump_is_frozen() {
|
||
let sample = |i: usize| {
|
||
group_sample(&[
|
||
("CPU Energy", if i < 3 { 480_083_160 } else { 541_893_638 }),
|
||
("PCPU", 7 * i as i64),
|
||
("GPU Energy", 1_000 + 7 * i as i64),
|
||
])
|
||
};
|
||
let f = fake(true, TICK, |i| Some(sample(i)));
|
||
assert_eq!(f.run.verdict(), Some(CpuRail::Frozen), "one refresh counted as a live rail: {:?}", f.run.watch);
|
||
assert_eq!(status_of(f.run.verdict()), crate::IoReportStatus::CpuRailFrozen);
|
||
assert_eq!((f.samples, f.ticks, f.run.ticks as usize, f.run.calls.len()), (41, 40, 40, 41));
|
||
for (i, call) in f.run.calls.iter().enumerate() {
|
||
assert_eq!(call.shape, Some((3, true)), "sample {i}");
|
||
let want = vec![
|
||
(key("CPU Energy", 0), if i < 3 { 480_083_160 } else { 541_893_638 }),
|
||
(key("GPU Energy", 0), 1_000 + 7 * i as i64),
|
||
];
|
||
assert_eq!(call.handed.as_ref(), Some(&want), "sample {i}: the record is not what the rule was handed");
|
||
// The injected clocks: the fake thread clock moves 1 ms in a
|
||
// sample call, the fake wall clock not at all.
|
||
assert_eq!(call.cpu_after - call.cpu_before, Duration::from_millis(1), "sample {i}");
|
||
assert_eq!(call.wall, Duration::ZERO, "sample {i}");
|
||
}
|
||
// Between one call's end and the next one's start, the fake thread
|
||
// clock moved by exactly the tick.
|
||
for w in f.run.calls.windows(2) {
|
||
assert_eq!(w[1].cpu_before - w[0].cpu_after, TICK);
|
||
}
|
||
}
|
||
|
||
/// The same CPU rail advancing in every interval, 25 ms of wall clock
|
||
/// apart: Live once the rule holds, at the 12th interval (300 ms), and
|
||
/// the probe stops right there. The window the rule saw is the
|
||
/// composition's own wall clock, from the first sample to the last.
|
||
#[test]
|
||
fn probe_with_a_sustained_advance_is_live() {
|
||
let f = fake(true, TICK, |i| Some(group_sample(&[("GPU Energy", 5), ("CPU Energy", 1_000 + 13 * i as i64)])));
|
||
assert_eq!(f.run.verdict(), Some(CpuRail::Live));
|
||
assert_eq!(status_of(f.run.verdict()), crate::IoReportStatus::Live);
|
||
assert_eq!((f.samples, f.ticks), (13, 12), "a live rail must stop the probe as soon as the rule holds");
|
||
assert_eq!((f.run.watch.intervals, f.run.watch.window), (12, Duration::from_millis(300)));
|
||
}
|
||
|
||
/// The observed two-step refresh through the whole composition, beside
|
||
/// a moving GPU rail: Frozen, after the whole budget.
|
||
#[test]
|
||
fn probe_with_the_two_step_refresh_is_frozen() {
|
||
let f = fake(true, TICK, |i| {
|
||
Some(group_sample(&[("CPU Energy", two_step_refresh(i, 3, 4)), ("GPU Energy", 1_000 + 7 * i as i64)]))
|
||
});
|
||
assert_eq!(f.run.verdict(), Some(CpuRail::Frozen), "the two-step refresh counted as live: {:?}", f.run.watch);
|
||
assert_eq!(status_of(f.run.verdict()), crate::IoReportStatus::CpuRailFrozen);
|
||
assert_eq!(f.run.watch.rails, vec![(key("CPU Energy", 0), 2)]);
|
||
assert_eq!((f.samples, f.ticks), (41, 40));
|
||
}
|
||
|
||
/// Two dies, each stepping once, in different intervals: Frozen, and
|
||
/// each die's own value is what the rule was handed. A transform that
|
||
/// gave each CPU rail the sum of the CPU rails would step both "rails"
|
||
/// in both intervals and call them live.
|
||
#[test]
|
||
fn probe_with_one_step_on_each_die_in_different_intervals_is_frozen() {
|
||
let die = |i: usize| {
|
||
[
|
||
("DIE_0_CPU Energy", if i >= 3 { 9_500_000 } else { 9_494_336 }),
|
||
("DIE_1_CPU Energy", if i >= 6 { 10_300_000 } else { 10_212_656 }),
|
||
]
|
||
};
|
||
let f = fake(true, TICK, |i| Some(group_sample(&die(i))));
|
||
assert_eq!(f.run.verdict(), Some(CpuRail::Frozen), "one step per die counted as a live rail: {:?}", f.run.watch);
|
||
assert_eq!(f.run.watch.rails, vec![(key("DIE_0_CPU Energy", 0), 1), (key("DIE_1_CPU Energy", 0), 1)]);
|
||
for (i, call) in f.run.calls.iter().enumerate() {
|
||
let [(n0, v0), (n1, v1)] = die(i);
|
||
assert_eq!(call.handed, Some(vec![(key(n0, 0), v0), (key(n1, 0), v1)]), "sample {i}");
|
||
}
|
||
}
|
||
|
||
/// One die live, one frozen, through the whole composition: Frozen,
|
||
/// after the whole budget, with both dies' own values handed to the
|
||
/// rule. Both live: Live, where one die alone is.
|
||
#[test]
|
||
fn probe_with_one_die_live_and_one_frozen_is_frozen() {
|
||
let die = |i: usize, die1_live: bool| {
|
||
[
|
||
("DIE_0_CPU Energy", 9_494_336 + 1_307 * i as i64),
|
||
("DIE_1_CPU Energy", 10_212_656 + if die1_live { 1_211 * i as i64 } else { 0 }),
|
||
]
|
||
};
|
||
let f = fake(true, TICK, |i| Some(group_sample(&die(i, false))));
|
||
assert_eq!(f.run.verdict(), Some(CpuRail::Frozen), "one die live, one frozen, counted as live: {:?}", f.run.watch);
|
||
assert_eq!(status_of(f.run.verdict()), crate::IoReportStatus::CpuRailFrozen);
|
||
assert_eq!(f.run.watch.rails, vec![(key("DIE_0_CPU Energy", 0), 40), (key("DIE_1_CPU Energy", 0), 0)]);
|
||
assert_eq!((f.samples, f.ticks), (41, 40));
|
||
for (i, call) in f.run.calls.iter().enumerate() {
|
||
let [(n0, v0), (n1, v1)] = die(i, false);
|
||
assert_eq!(call.handed, Some(vec![(key(n0, 0), v0), (key(n1, 0), v1)]), "sample {i}");
|
||
}
|
||
|
||
let f = fake(true, TICK, |i| Some(group_sample(&die(i, true))));
|
||
assert_eq!(f.run.verdict(), Some(CpuRail::Live), "{:?}", f.run.watch);
|
||
assert_eq!(status_of(f.run.verdict()), crate::IoReportStatus::Live);
|
||
assert_eq!((f.samples, f.ticks), (13, 12));
|
||
}
|
||
|
||
/// A clock pair for [`spin_for_cpu`] that only the spin moves: each read
|
||
/// of the CPU clock advances it by `cpu_step`, and each read of the wall
|
||
/// clock advances it by `wall_step`. The spin reads each once per turn,
|
||
/// so a turn is `cpu_step` of CPU in `wall_step` of wall time; and a
|
||
/// spin that ignores one clock still ends on the other. Returns the CPU
|
||
/// the spin reported and how many times it read the CPU clock.
|
||
fn spin_scripted(cpu_step: Duration, wall_step: Duration, want: Duration, cap: Duration) -> (Duration, u32) {
|
||
let (cpu_reads, wall_reads) = (Cell::new(0u32), Cell::new(0u32));
|
||
let used = spin_for_cpu(
|
||
|| {
|
||
assert!(cpu_reads.get() < 1_000_000, "the tick never stopped");
|
||
cpu_reads.set(cpu_reads.get() + 1);
|
||
cpu_step * cpu_reads.get()
|
||
},
|
||
|| {
|
||
assert!(wall_reads.get() < 1_000_000, "the tick never stopped");
|
||
wall_reads.set(wall_reads.get() + 1);
|
||
wall_step * wall_reads.get()
|
||
},
|
||
want,
|
||
cap,
|
||
);
|
||
(used, cpu_reads.get())
|
||
}
|
||
|
||
/// The tick spins until this thread's CPU clock has advanced by the
|
||
/// tick, not the wall clock: a thread that gets a quarter of a core
|
||
/// still burns the whole 25 ms (in 100 ms of wall time). Only the wall
|
||
/// cap ends a tick early: a thread that never runs, or gets under 5% of
|
||
/// a core, stops at 500 ms of wall time.
|
||
#[test]
|
||
fn the_tick_spins_until_the_thread_cpu_clock_advanced() {
|
||
let ms = Duration::from_millis;
|
||
let (want, cap) = (ms(25), ms(500));
|
||
// A quarter of a core: 25 ms of CPU, 100 ms of wall time.
|
||
let (used, steps) = spin_scripted(ms(1), ms(4), want, cap);
|
||
assert_eq!((used, steps), (ms(25), 26), "the tick must burn 25 ms of thread CPU, however slowly it gets it");
|
||
// A whole core.
|
||
let (used, _) = spin_scripted(ms(1), ms(1), want, cap);
|
||
assert_eq!(used, ms(25));
|
||
// Never scheduled: the wall cap ends it with no CPU burned.
|
||
let (used, steps) = spin_scripted(Duration::ZERO, ms(10), want, cap);
|
||
assert_eq!((used, steps), (Duration::ZERO, 51), "the wall cap must end a tick that gets no CPU");
|
||
// 2.5% of a core: the wall cap, at 13 ms of CPU.
|
||
let (used, steps) = spin_scripted(ms(1), ms(40), want, cap);
|
||
assert_eq!((used, steps), (ms(13), 14));
|
||
}
|
||
|
||
/// Only a live CPU rail admits the counter.
|
||
#[test]
|
||
fn only_a_live_rail_admits_the_counter() {
|
||
use crate::IoReportStatus as S;
|
||
assert!(admits_counter(S::Live));
|
||
for declined in [S::Unavailable, S::CpuRailAbsent, S::CpuRailFrozen] {
|
||
assert!(!admits_counter(declined), "{declined:?} admitted the counter");
|
||
}
|
||
}
|
||
|
||
// ── A single vendor counter (RAPL, NVML) ─────────────────────────────
|
||
|
||
/// What a scripted counter probe did: its status, how many samples it
|
||
/// took and how many ticks it ran. `opens` is whether the counter
|
||
/// opens; `read(i)` is its i-th sample, `None` for a failed read. The
|
||
/// clock moves only with the tick, by `step`.
|
||
fn counter_probe(opens: bool, step: Duration, mut read: impl FnMut(usize) -> Option<i64>) -> (crate::CounterStatus, usize, u32) {
|
||
let clock = Cell::new(Duration::ZERO);
|
||
let (mut taken, mut ticks) = (0usize, 0u32);
|
||
let probed = probe_counter(
|
||
|| opens.then_some("counter"),
|
||
|c: &&str| {
|
||
assert!(taken < 100_000, "the probe never stopped sampling");
|
||
taken += 1;
|
||
read(taken - 1).map(|raw| vec![(c.to_string(), raw)])
|
||
},
|
||
|| {
|
||
ticks += 1;
|
||
clock.set(clock.get() + step);
|
||
},
|
||
|| clock.get(),
|
||
BUDGET,
|
||
);
|
||
(counter_status(probed.as_ref()), taken, ticks)
|
||
}
|
||
|
||
/// A counter that advances at every sample is live once five intervals
|
||
/// and 300 ms have been watched: at 25 ms a sample, 12 intervals.
|
||
#[test]
|
||
fn a_vendor_counter_advancing_every_sample_is_live() {
|
||
let (status, taken, ticks) = counter_probe(true, TICK, |i| Some(81_000_000 + 1_400 * i as i64));
|
||
assert_eq!((status, taken, ticks), (crate::CounterStatus::Live, 13, 12));
|
||
// Slow samples: five intervals of 100 ms are 500 ms, past the window.
|
||
let (status, taken, _) = counter_probe(true, Duration::from_millis(100), |i| Some(i as i64));
|
||
assert_eq!((status, taken), (crate::CounterStatus::Live, 6));
|
||
}
|
||
|
||
/// A counter that holds is frozen, watched for the whole budget.
|
||
#[test]
|
||
fn a_vendor_counter_that_holds_is_frozen() {
|
||
let (status, taken, _) = counter_probe(true, TICK, |_| Some(81_000_000));
|
||
assert_eq!((status, taken), (crate::CounterStatus::Frozen, 41));
|
||
}
|
||
|
||
/// A counter republished in lumps — one step, or a refresh in two
|
||
/// steps 64 ms apart — is frozen: never the sustained advance of a live
|
||
/// count, and a lump landing in one caller's span would be sealed as
|
||
/// that span's reading.
|
||
#[test]
|
||
fn a_vendor_counter_republished_in_lumps_is_frozen() {
|
||
let (status, ..) = counter_probe(true, TICK, |i| Some(if i < 7 { 1_000 } else { 61_000 }));
|
||
assert_eq!(status, crate::CounterStatus::Frozen, "one step");
|
||
let (status, ..) = counter_probe(true, Duration::from_millis(32), |i| Some(two_step_refresh(i, 3, 5)));
|
||
assert_eq!(status, crate::CounterStatus::Frozen, "a two-step refresh");
|
||
// Advancing in two of every three intervals is not a live count...
|
||
let (status, ..) = counter_probe(true, TICK, |i| Some((i - i / 3) as i64));
|
||
assert_eq!(status, crate::CounterStatus::Frozen, "advancing in 2 of every 3 intervals");
|
||
// ...and a counter missing one interval in five still is live.
|
||
let (status, ..) = counter_probe(true, TICK, |i| Some((i - i / 5) as i64));
|
||
assert_eq!(status, crate::CounterStatus::Live, "advancing in 4 of every 5 intervals");
|
||
}
|
||
|
||
/// A counter that wraps (RAPL's `energy_uj` rolls over at
|
||
/// `max_energy_range_uj`) fell over that interval, which is no advance;
|
||
/// it is live again from the next one.
|
||
#[test]
|
||
fn a_wrap_is_not_an_advance() {
|
||
let (status, ..) = counter_probe(true, TICK, |i| Some(if i < 3 { 262_000_000 + i as i64 } else { i as i64 }));
|
||
assert_eq!(status, crate::CounterStatus::Live);
|
||
let w = {
|
||
let mut w = RailWatch::every_channel();
|
||
for (i, raw) in [(0, 900), (1, 950), (2, 10), (3, 60)] {
|
||
w.observe(&[("intel-rapl:0".to_string(), raw)], TICK * i);
|
||
}
|
||
w
|
||
};
|
||
assert_eq!(w.rails, vec![(key("intel-rapl:0", 0), 2)], "the fall is no advance: {w:?}");
|
||
}
|
||
|
||
/// A counter that opens but never reads is sampled to the cap of five
|
||
/// budgets and is unavailable; one that does not open is unavailable
|
||
/// with nothing sampled and nothing ticked.
|
||
#[test]
|
||
fn a_vendor_counter_that_never_reads_or_opens_is_unavailable() {
|
||
let (status, taken, _) = counter_probe(true, TICK, |_| None);
|
||
assert_eq!((status, taken), (crate::CounterStatus::Unavailable, 201));
|
||
let (status, taken, ticks) = counter_probe(false, TICK, |_| panic!("sampled a counter that did not open"));
|
||
assert_eq!((status, taken, ticks), (crate::CounterStatus::Unavailable, 0, 0));
|
||
}
|
||
|
||
/// Every channel of a vendor counter's sample is a rail; IOReport's
|
||
/// watch keeps only the CPU energy aggregates.
|
||
#[test]
|
||
fn a_vendor_counter_watches_every_channel() {
|
||
let sample = |i: i64| vec![("intel-rapl:0".to_string(), 10 * i), ("CPU Energy".to_string(), 5)];
|
||
let (mut all, mut cpu) = (RailWatch::every_channel(), RailWatch::default());
|
||
for i in 0..3 {
|
||
all.observe(&sample(i), TICK * i as u32);
|
||
cpu.observe(&sample(i), TICK * i as u32);
|
||
}
|
||
assert_eq!(all.rails, vec![(key("intel-rapl:0", 0), 2), (key("CPU Energy", 0), 0)]);
|
||
assert_eq!(cpu.rails, vec![(key("CPU Energy", 0), 0)]);
|
||
}
|
||
|
||
/// Only a live counter admits its rung.
|
||
#[test]
|
||
fn only_a_live_vendor_counter_admits_its_rung() {
|
||
use crate::CounterStatus as S;
|
||
assert!(admits_vendor_counter(S::Live));
|
||
for declined in [S::Unavailable, S::Frozen] {
|
||
assert!(!admits_vendor_counter(declined), "{declined:?} admitted the rung");
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
/// The clocks and the tick of the liveness probes that burn CPU (IOReport's,
|
||
/// RAPL's): this thread's CPU clock, a monotonic wall clock, and a tick that
|
||
/// spins until this thread has burned [`clocks::PROBE_TICK`] of its own CPU.
|
||
/// The one other module allowed `unsafe`: the thread CPU clock is only
|
||
/// reachable through `clock_gettime`.
|
||
#[cfg(any(target_os = "macos", target_os = "linux"))]
|
||
#[allow(unsafe_code)]
|
||
mod clocks {
|
||
use std::os::raw::{c_int, c_long};
|
||
use std::sync::OnceLock;
|
||
use std::time::{Duration, Instant};
|
||
|
||
/// `struct timespec` on 64-bit macOS and Linux.
|
||
#[repr(C)]
|
||
struct Timespec {
|
||
tv_sec: c_long,
|
||
tv_nsec: c_long,
|
||
}
|
||
|
||
/// `clockid_t`: an unsigned enum on macOS, an `int` on Linux.
|
||
#[cfg(target_os = "macos")]
|
||
type ClockId = std::os::raw::c_uint;
|
||
#[cfg(target_os = "linux")]
|
||
type ClockId = c_int;
|
||
|
||
/// `CLOCK_THREAD_CPUTIME_ID`: 16 in macOS `<time.h>`, 3 in Linux
|
||
/// `<linux/time.h>`.
|
||
#[cfg(target_os = "macos")]
|
||
const CLOCK_THREAD_CPUTIME_ID: ClockId = 16;
|
||
#[cfg(target_os = "linux")]
|
||
const CLOCK_THREAD_CPUTIME_ID: ClockId = 3;
|
||
|
||
unsafe extern "C" {
|
||
fn clock_gettime(clock_id: ClockId, tp: *mut Timespec) -> c_int;
|
||
}
|
||
|
||
/// CPU time the calling thread has consumed. Zero if the clock cannot be
|
||
/// read, which leaves a probe tick spinning to its wall cap: slower, but
|
||
/// still burning CPU.
|
||
pub(crate) fn thread_cpu_time() -> Duration {
|
||
let mut ts = Timespec { tv_sec: 0, tv_nsec: 0 };
|
||
if unsafe { clock_gettime(CLOCK_THREAD_CPUTIME_ID, &mut ts) } != 0 {
|
||
return Duration::ZERO;
|
||
}
|
||
Duration::new(ts.tv_sec.max(0) as u64, ts.tv_nsec.clamp(0, 999_999_999) as u32)
|
||
}
|
||
|
||
/// Wall time on the monotonic clock, from a fixed origin in this process.
|
||
pub(crate) fn wall_now() -> Duration {
|
||
static ORIGIN: OnceLock<Instant> = OnceLock::new();
|
||
ORIGIN.get_or_init(Instant::now).elapsed()
|
||
}
|
||
|
||
/// The CPU the probing thread burns between two liveness samples: this
|
||
/// much of its own CPU time, measured on its CPU clock, so host load
|
||
/// stretches a tick's wall time but never shortens its CPU.
|
||
pub(crate) const PROBE_TICK: Duration = Duration::from_millis(25);
|
||
|
||
/// However little CPU the probing thread gets, one tick ends after this
|
||
/// much wall time.
|
||
pub(crate) const PROBE_TICK_WALL_CAP: Duration = Duration::from_millis(500);
|
||
|
||
/// One tick of a CPU-burning probe: spin until this thread's CPU clock has
|
||
/// advanced [`PROBE_TICK`], or [`PROBE_TICK_WALL_CAP`] of wall time has
|
||
/// passed.
|
||
pub(crate) fn cpu_burn_tick() {
|
||
crate::rail_liveness::spin_for_cpu(thread_cpu_time, wall_now, PROBE_TICK, PROBE_TICK_WALL_CAP);
|
||
}
|
||
}
|
||
|
||
// Apple Silicon IOReport energy counter — a no-sudo, no-external-binary read of
|
||
// the SoC Energy Model channels through Apple's private IOReport C API. Ported
|
||
// from jouleclaw-energy's `apple_ioreport` backend, adapted to eoc's
|
||
// `JouleCounter`.
|
||
//
|
||
// An open subscription is not evidence that the counters are live. On macOS
|
||
// 27.0.1, for an unprivileged reader, the CPU rails (`CPU Energy`, `PCPU`,
|
||
// `MCPU*`, `PACC_*`, `DIE_<n>_CPU Energy`) hold their value under full CPU load
|
||
// on both an M5 Max and an M3 Ultra, and are refreshed only now and then (an M3
|
||
// Ultra's held for at least 22 minutes, then stepped by ~1.3 kJ per die in one
|
||
// sample; an M5 Max's `CPU Energy` has been seen to publish one refresh in two
|
||
// steps 64 ms apart), while `GPU Energy` still moves (steadily on the M5 Max,
|
||
// intermittently on the M3 Ultra). Summed, the rails report GPU-only energy,
|
||
// zero, or a refresh of past CPU energy, and consumers such as eoc-cascade,
|
||
// eoc-local and eoc-streaming tag what a hardware counter returns as Measured.
|
||
// So the counter is constructed only when `ioreport_status()` is `Live` — every
|
||
// CPU rail in the samples (each die's, on a multi-die part) kept advancing while
|
||
// the probing thread burned CPU: each in at least five separate sample
|
||
// intervals, at least 80% of those observed, over at least 300 ms — and
|
||
// otherwise `try_detect()` returns `Err(NoCounter)`, leaving the caller's own
|
||
// estimate. Every rail, because the counter sums them all.
|
||
#[cfg(target_os = "macos")]
|
||
#[allow(non_camel_case_types, unsafe_code)]
|
||
mod macos_ioreport {
|
||
use super::*;
|
||
use crate::clocks::{thread_cpu_time, wall_now};
|
||
use crate::rail_liveness::{self, ProbeRun, Sample};
|
||
use std::os::raw::{c_int, c_long, c_void};
|
||
use std::sync::Mutex;
|
||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||
|
||
type CFTypeRef = *const c_void;
|
||
type CFStringRef = *const c_void;
|
||
type CFArrayRef = *const c_void;
|
||
type CFDictionaryRef = *const c_void;
|
||
type CFMutableDictionaryRef = *mut c_void;
|
||
type IOReportSubscriptionRef = *const c_void;
|
||
const K_CFSTRING_ENCODING_UTF8: u32 = 0x0800_0100;
|
||
|
||
unsafe extern "C" {
|
||
fn IOReportCopyAllChannels(a: u64, b: u64) -> CFMutableDictionaryRef;
|
||
fn IOReportCopyChannelsInGroup(
|
||
group: CFStringRef,
|
||
subgroup: CFStringRef,
|
||
a: u64,
|
||
b: u64,
|
||
c: u64,
|
||
) -> CFMutableDictionaryRef;
|
||
fn IOReportCreateSubscription(
|
||
allowed: *const c_void,
|
||
desired_channels: CFMutableDictionaryRef,
|
||
subscribed_channels: *mut CFMutableDictionaryRef,
|
||
channel_id: u64,
|
||
options: CFTypeRef,
|
||
) -> IOReportSubscriptionRef;
|
||
fn IOReportCreateSamples(
|
||
iorsub: IOReportSubscriptionRef,
|
||
chan_dict: CFMutableDictionaryRef,
|
||
options: CFTypeRef,
|
||
) -> CFDictionaryRef;
|
||
fn IOReportCreateSamplesDelta(
|
||
previous: CFDictionaryRef,
|
||
current: CFDictionaryRef,
|
||
options: CFTypeRef,
|
||
) -> CFDictionaryRef;
|
||
fn IOReportChannelGetGroup(item: CFTypeRef) -> CFStringRef;
|
||
fn IOReportChannelGetChannelName(item: CFTypeRef) -> CFStringRef;
|
||
fn IOReportSimpleGetIntegerValue(item: CFTypeRef, _v: c_int) -> c_long;
|
||
fn IOReportChannelGetUnitLabel(item: CFTypeRef) -> CFStringRef;
|
||
fn CFRelease(cf: CFTypeRef);
|
||
fn CFArrayGetCount(the_array: CFArrayRef) -> c_long;
|
||
fn CFArrayGetValueAtIndex(the_array: CFArrayRef, idx: c_long) -> CFTypeRef;
|
||
fn CFDictionaryGetValue(the_dict: CFDictionaryRef, key: *const c_void) -> *const c_void;
|
||
fn CFStringCreateWithCString(
|
||
allocator: *mut c_void,
|
||
cstr: *const i8,
|
||
encoding: u32,
|
||
) -> CFStringRef;
|
||
fn CFStringGetCString(
|
||
the_string: CFStringRef,
|
||
buffer: *mut i8,
|
||
buffer_size: c_long,
|
||
encoding: u32,
|
||
) -> bool;
|
||
}
|
||
|
||
// Three distinct frameworks, each legitimately `kind = "framework"`; the
|
||
// lint mis-reads the repeated key across separate `#[link]`s as a duplicate.
|
||
#[allow(clippy::duplicated_attributes)]
|
||
#[link(name = "IOReport", kind = "dylib")]
|
||
#[link(name = "IOKit", kind = "framework")]
|
||
#[link(name = "CoreFoundation", kind = "framework")]
|
||
unsafe extern "C" {}
|
||
|
||
struct State {
|
||
sub: IOReportSubscriptionRef,
|
||
channels: CFMutableDictionaryRef,
|
||
prior_sample: Option<CFDictionaryRef>,
|
||
cumulative_uj: u64,
|
||
}
|
||
|
||
impl Drop for State {
|
||
fn drop(&mut self) {
|
||
unsafe {
|
||
if let Some(prior) = self.prior_sample.take() {
|
||
CFRelease(prior);
|
||
}
|
||
if !self.sub.is_null() {
|
||
CFRelease(self.sub);
|
||
}
|
||
if !self.channels.is_null() {
|
||
CFRelease(self.channels as CFTypeRef);
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
/// Apple Silicon IOReport energy counter (SoC-total).
|
||
///
|
||
/// Constructed only on a host whose every CPU energy rail is live; see
|
||
/// [`crate::ioreport_status`]. A live IOReport reading is a vendor
|
||
/// energy counter's reading, so the delta it gives is `Measured`, as
|
||
/// RAPL's and NVML's are; that holds only while the rail is live, which
|
||
/// is why the counter is never constructed when it is not.
|
||
pub struct MacosIoReportCounter {
|
||
state: Mutex<State>,
|
||
}
|
||
|
||
// Raw CF pointers are confined to `State` behind a `Mutex`.
|
||
unsafe impl Send for MacosIoReportCounter {}
|
||
unsafe impl Sync for MacosIoReportCounter {}
|
||
|
||
/// Subscribe to `desired` (consumed). `None`, with everything released,
|
||
/// if IOReport returns no subscription or no subscribed-channel dictionary.
|
||
unsafe fn subscribe(desired: CFMutableDictionaryRef) -> Option<State> {
|
||
unsafe {
|
||
let mut subscribed: CFMutableDictionaryRef = std::ptr::null_mut();
|
||
let sub = IOReportCreateSubscription(
|
||
std::ptr::null(),
|
||
desired,
|
||
&mut subscribed as *mut _,
|
||
0,
|
||
std::ptr::null(),
|
||
);
|
||
CFRelease(desired as CFTypeRef);
|
||
if sub.is_null() || subscribed.is_null() {
|
||
if !sub.is_null() {
|
||
CFRelease(sub);
|
||
}
|
||
if !subscribed.is_null() {
|
||
CFRelease(subscribed as CFTypeRef);
|
||
}
|
||
return None;
|
||
}
|
||
Some(State { sub, channels: subscribed, prior_sample: None, cumulative_uj: 0 })
|
||
}
|
||
}
|
||
|
||
impl MacosIoReportCounter {
|
||
/// Open an IOReport subscription on a host whose CPU energy rail is
|
||
/// live. `Err(NoCounter)` if IOReport can't open (non-Apple-Silicon,
|
||
/// Rosetta, or a future macOS that changes the API), **and** if it opens
|
||
/// but its CPU energy rail is absent, or does not keep advancing under
|
||
/// load — then the summed rails would report GPU-only energy, zero, or a
|
||
/// stale refresh as the energy of CPU work. [`crate::ioreport_status`]
|
||
/// says which.
|
||
pub fn try_detect() -> Result<Self> {
|
||
Self::try_detect_given(crate::ioreport_status())
|
||
}
|
||
|
||
/// The constructor behind [`Self::try_detect`], given this host's
|
||
/// status: every status the pure gate does not admit declines before
|
||
/// anything opens. Opening a subscription is not enough; the CPU rail
|
||
/// must be live.
|
||
pub(crate) fn try_detect_given(status: crate::IoReportStatus) -> Result<Self> {
|
||
if !rail_liveness::admits_counter(status) {
|
||
return Err(Error::NoCounter);
|
||
}
|
||
match Self::open() {
|
||
Some(state) => Ok(Self { state: Mutex::new(state) }),
|
||
None => Err(Error::NoCounter),
|
||
}
|
||
}
|
||
|
||
/// The counter's own subscription: every channel.
|
||
fn open() -> Option<State> {
|
||
// Test builds only: count the attempt on this thread, so a test can
|
||
// pin that a declined status never gets this far.
|
||
#[cfg(test)]
|
||
script::note_counter_open();
|
||
unsafe {
|
||
let all = IOReportCopyAllChannels(0, 0);
|
||
if all.is_null() {
|
||
return None;
|
||
}
|
||
subscribe(all)
|
||
}
|
||
}
|
||
|
||
/// A subscription to the Energy Model group alone, for the liveness
|
||
/// probe. The CPU rails are the same counters the all-channel
|
||
/// subscription above reads, with the same absolute values, but one
|
||
/// sample of this group takes ~6-12 ms on an M5 Max and an M3 Ultra,
|
||
/// against 0.13-0.8 s and ~0.14 s for every channel — slow enough that a
|
||
/// probe sampling all channels would spend nearly its whole budget
|
||
/// inside IOReport instead of burning CPU. On an M5 Max a probe on
|
||
/// every channel saw 2-3 intervals in ~2 s; on this group, 25-31 in one.
|
||
fn open_energy_model() -> Option<State> {
|
||
// Test builds only: a script installed on this thread stands in for
|
||
// IOReport here and in `State::group_sample`, and nowhere else.
|
||
#[cfg(test)]
|
||
if let Some(opens) = script::opens() {
|
||
return opens.then(State::detached);
|
||
}
|
||
unsafe {
|
||
let group = CFStringCreateWithCString(
|
||
std::ptr::null_mut(),
|
||
c"Energy Model".as_ptr() as *const _,
|
||
K_CFSTRING_ENCODING_UTF8,
|
||
);
|
||
if group.is_null() {
|
||
return None;
|
||
}
|
||
let desired = IOReportCopyChannelsInGroup(group, std::ptr::null(), 0, 0, 0);
|
||
CFRelease(group);
|
||
if desired.is_null() {
|
||
return None;
|
||
}
|
||
subscribe(desired)
|
||
}
|
||
}
|
||
}
|
||
|
||
impl State {
|
||
/// A state that holds no subscription (its drop releases nothing), for
|
||
/// a script to sample in place of IOReport.
|
||
#[cfg(test)]
|
||
fn detached() -> State {
|
||
State {
|
||
sub: std::ptr::null(),
|
||
channels: std::ptr::null_mut(),
|
||
prior_sample: None,
|
||
cumulative_uj: 0,
|
||
}
|
||
}
|
||
|
||
/// One sample of this subscription: how many channels it held, whether
|
||
/// every one is in the Energy Model group, and each Energy Model
|
||
/// channel's (name, absolute raw integer), in sample order. `None` when
|
||
/// IOReport returns no sample, or a sample with no channel list. This
|
||
/// only extracts; the liveness rule lives in [`rail_liveness`].
|
||
fn group_sample(&self) -> Option<Sample> {
|
||
// Test builds only: a script installed on this thread stands in for
|
||
// IOReport here and in `open_energy_model`, and nowhere else.
|
||
#[cfg(test)]
|
||
if let Some(sample) = script::next() {
|
||
return sample;
|
||
}
|
||
unsafe {
|
||
let smp = IOReportCreateSamples(self.sub, self.channels, std::ptr::null());
|
||
if smp.is_null() {
|
||
return None;
|
||
}
|
||
let sample = energy_model_channels(smp);
|
||
CFRelease(smp);
|
||
sample
|
||
}
|
||
}
|
||
}
|
||
|
||
/// In a sample (absolute, not a delta): its channel count, whether every
|
||
/// channel is in the Energy Model group (by `IOReportChannelGetGroup`), and
|
||
/// each Energy Model channel's name and raw integer value. `None` when the
|
||
/// sample carries no channel list.
|
||
fn energy_model_channels(sample: CFDictionaryRef) -> Option<Sample> {
|
||
unsafe {
|
||
let key = CFStringCreateWithCString(
|
||
std::ptr::null_mut(),
|
||
c"IOReportChannels".as_ptr() as *const _,
|
||
K_CFSTRING_ENCODING_UTF8,
|
||
);
|
||
if key.is_null() {
|
||
return None;
|
||
}
|
||
let arr = CFDictionaryGetValue(sample, key as *const _) as CFArrayRef;
|
||
CFRelease(key as CFTypeRef);
|
||
if arr.is_null() {
|
||
return None;
|
||
}
|
||
let count = CFArrayGetCount(arr);
|
||
let (mut group_only, mut values) = (true, Vec::new());
|
||
for i in 0..count {
|
||
let item = CFArrayGetValueAtIndex(arr, i);
|
||
if item.is_null() || !cfstring_eq_ic(IOReportChannelGetGroup(item), "energy model") {
|
||
group_only = false;
|
||
continue;
|
||
}
|
||
let Some(name) = cfstring_to_string(IOReportChannelGetChannelName(item)) else {
|
||
continue;
|
||
};
|
||
values.push((name, IOReportSimpleGetIntegerValue(item, 0) as i64));
|
||
}
|
||
Some(Sample { channels: count.max(0) as usize, group_only, values })
|
||
}
|
||
}
|
||
|
||
/// One tick of the probe: [`crate::clocks::cpu_burn_tick`].
|
||
fn probe_tick() {
|
||
crate::clocks::cpu_burn_tick();
|
||
}
|
||
|
||
/// How many times [`host_status`] — the probe behind the cached
|
||
/// [`crate::ioreport_status`] — has run in this process, on any thread.
|
||
pub(crate) static HOST_STATUS_RUNS: AtomicUsize = AtomicUsize::new(0);
|
||
|
||
/// How many times this process has opened the liveness probe's sampler,
|
||
/// on any thread: every real probe opens one, whoever calls it.
|
||
pub(crate) static SAMPLER_OPENS: AtomicUsize = AtomicUsize::new(0);
|
||
|
||
/// The real opener: a fresh subscription to the Energy Model group, and the
|
||
/// sampler that reads it. The probe takes its own samples, so no counter's
|
||
/// cumulative reading is touched; the subscription is dropped with the
|
||
/// sampler, when the probe ends.
|
||
fn open_group_sampler() -> Option<impl FnMut() -> Option<Sample>> {
|
||
SAMPLER_OPENS.fetch_add(1, Ordering::SeqCst);
|
||
let state = MacosIoReportCounter::open_energy_model()?;
|
||
Some(move || state.group_sample())
|
||
}
|
||
|
||
/// The production probe: [`rail_liveness::probe_with`] on the real opener,
|
||
/// the real tick, this thread's CPU clock and the monotonic wall clock, for
|
||
/// [`crate::CPU_RAIL_LIVENESS_BUDGET`]. The only production caller of
|
||
/// `probe_with`, called in production only by [`host_status`]; the
|
||
/// hardware tests run it too, uncounted and uncached, to pin what the
|
||
/// production composition hands the rule.
|
||
pub(crate) fn production_probe() -> ProbeRun {
|
||
rail_liveness::probe_with(open_group_sampler, probe_tick, thread_cpu_time, wall_now, crate::CPU_RAIL_LIVENESS_BUDGET)
|
||
}
|
||
|
||
/// This host's status: the pure mapping of the production probe's verdict.
|
||
/// Counts its own runs first; cached, once per process, by
|
||
/// [`crate::ioreport_status`], its only production caller.
|
||
pub(crate) fn host_status() -> crate::IoReportStatus {
|
||
HOST_STATUS_RUNS.fetch_add(1, Ordering::SeqCst);
|
||
rail_liveness::status_of(production_probe().verdict())
|
||
}
|
||
|
||
/// Test support: a scripted sample stream that stands in for IOReport in
|
||
/// the probe's two FFI leaves — opening the Energy Model group
|
||
/// subscription, and taking one sample of it — on the thread that installs
|
||
/// it. Everything above those leaves runs as in production: the counted
|
||
/// [`host_status`], [`production_probe`] with the real tick and clocks, the
|
||
/// real opener's sampler closure, the rule and the status mapping. So a
|
||
/// test can run the production status itself with no subscription, a
|
||
/// batch refresh, or a live rail, none of which these hosts produce on
|
||
/// demand. Likewise a status installed on a thread stands in for the
|
||
/// cached one in [`crate::ioreport_status`], so the real constructor and
|
||
/// selection run under every status; and each thread counts its attempts
|
||
/// to open the counter's own all-channel subscription.
|
||
#[cfg(test)]
|
||
pub(crate) mod script {
|
||
use super::Sample;
|
||
use std::cell::RefCell;
|
||
|
||
/// What the leaves return while a script is installed.
|
||
pub(crate) struct Script {
|
||
/// Whether the group subscription opens.
|
||
pub(crate) opens: bool,
|
||
/// The i-th sample; `None` for one that cannot be taken.
|
||
pub(crate) samples: Box<dyn FnMut(usize) -> Option<Sample>>,
|
||
}
|
||
|
||
thread_local! {
|
||
static SCRIPT: RefCell<Option<(Script, usize)>> = const { RefCell::new(None) };
|
||
static STATUS: std::cell::Cell<Option<crate::IoReportStatus>> = const { std::cell::Cell::new(None) };
|
||
static COUNTER_OPENS: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
|
||
}
|
||
|
||
/// Called by the counter's own opener on every attempt to open its
|
||
/// all-channel subscription.
|
||
pub(crate) fn note_counter_open() {
|
||
COUNTER_OPENS.with(|c| c.set(c.get() + 1));
|
||
}
|
||
|
||
/// How many times this thread has tried to open the counter's
|
||
/// all-channel subscription. Per thread, so tests running beside it
|
||
/// that open one do not move it.
|
||
pub(crate) fn counter_opens() -> usize {
|
||
COUNTER_OPENS.with(|c| c.get())
|
||
}
|
||
|
||
/// `Some(status)` while a test has installed a status for
|
||
/// [`crate::ioreport_status`] to return on this thread.
|
||
pub(crate) fn status() -> Option<crate::IoReportStatus> {
|
||
STATUS.with(|s| s.get())
|
||
}
|
||
|
||
/// Run `f` on this thread with [`crate::ioreport_status`] returning
|
||
/// `status`, so the real `try_detect()` and `detect()` can be run
|
||
/// under every status. Removed even if `f` panics.
|
||
pub(crate) fn with_status<R>(status: crate::IoReportStatus, f: impl FnOnce() -> R) -> R {
|
||
struct Remove;
|
||
impl Drop for Remove {
|
||
fn drop(&mut self) {
|
||
STATUS.with(|s| s.set(None));
|
||
}
|
||
}
|
||
STATUS.with(|s| s.set(Some(status)));
|
||
let _remove = Remove;
|
||
f()
|
||
}
|
||
|
||
/// `Some(whether the subscription opens)` while a script is installed.
|
||
pub(crate) fn opens() -> Option<bool> {
|
||
SCRIPT.with(|s| s.borrow().as_ref().map(|(script, _)| script.opens))
|
||
}
|
||
|
||
/// `Some(the next scripted sample)` while a script is installed.
|
||
pub(crate) fn next() -> Option<Option<Sample>> {
|
||
SCRIPT.with(|s| {
|
||
s.borrow_mut().as_mut().map(|(script, served)| {
|
||
*served += 1;
|
||
(script.samples)(*served - 1)
|
||
})
|
||
})
|
||
}
|
||
|
||
/// Run `f` on this thread with `script` installed; returns what `f`
|
||
/// returned and how many samples the script served. The script is
|
||
/// removed even if `f` panics.
|
||
pub(crate) fn with<R>(script: Script, f: impl FnOnce() -> R) -> (R, usize) {
|
||
struct Remove;
|
||
impl Drop for Remove {
|
||
fn drop(&mut self) {
|
||
SCRIPT.with(|s| s.borrow_mut().take());
|
||
}
|
||
}
|
||
SCRIPT.with(|s| *s.borrow_mut() = Some((script, 0)));
|
||
let _remove = Remove;
|
||
let out = f();
|
||
let served = SCRIPT.with(|s| s.borrow().as_ref().map_or(0, |(_, served)| *served));
|
||
(out, served)
|
||
}
|
||
}
|
||
|
||
impl JouleCounter for MacosIoReportCounter {
|
||
fn read_microjoules(&self) -> Result<u64> {
|
||
let mut state = self
|
||
.state
|
||
.lock()
|
||
.map_err(|_| Error::Backend("IOReport state mutex poisoned".to_string()))?;
|
||
unsafe {
|
||
let current = IOReportCreateSamples(state.sub, state.channels, std::ptr::null());
|
||
if current.is_null() {
|
||
return Err(Error::Backend("IOReportCreateSamples returned null".to_string()));
|
||
}
|
||
if let Some(prior) = state.prior_sample {
|
||
let delta = IOReportCreateSamplesDelta(prior, current, std::ptr::null());
|
||
if !delta.is_null() {
|
||
state.cumulative_uj =
|
||
state.cumulative_uj.saturating_add(sum_energy_uj(delta));
|
||
CFRelease(delta);
|
||
}
|
||
CFRelease(prior);
|
||
}
|
||
state.prior_sample = Some(current);
|
||
}
|
||
Ok(state.cumulative_uj)
|
||
}
|
||
|
||
fn name(&self) -> &'static str {
|
||
"ioreport"
|
||
}
|
||
}
|
||
|
||
/// Sum every `Energy Model` channel in a samples-delta dict, in microjoules,
|
||
/// scaling each rail by its own unit label and dropping sub-rails of an
|
||
/// already-aggregated domain so energy is counted once.
|
||
fn sum_energy_uj(delta: CFDictionaryRef) -> u64 {
|
||
unsafe {
|
||
let key = CFStringCreateWithCString(
|
||
std::ptr::null_mut(),
|
||
c"IOReportChannels".as_ptr() as *const _,
|
||
K_CFSTRING_ENCODING_UTF8,
|
||
);
|
||
if key.is_null() {
|
||
return 0;
|
||
}
|
||
let arr = CFDictionaryGetValue(delta, key as *const _) as CFArrayRef;
|
||
CFRelease(key as CFTypeRef);
|
||
if arr.is_null() {
|
||
return 0;
|
||
}
|
||
let count = CFArrayGetCount(arr);
|
||
let mut total_uj: u64 = 0;
|
||
for i in 0..count {
|
||
let item = CFArrayGetValueAtIndex(arr, i);
|
||
if item.is_null() {
|
||
continue;
|
||
}
|
||
if !cfstring_eq_ic(IOReportChannelGetGroup(item), "energy model") {
|
||
continue;
|
||
}
|
||
let raw = IOReportSimpleGetIntegerValue(item, 0);
|
||
if raw <= 0 {
|
||
continue;
|
||
}
|
||
let name = match cfstring_to_string(IOReportChannelGetChannelName(item)) {
|
||
Some(s) => s,
|
||
None => continue,
|
||
};
|
||
if !counts_once(&name) {
|
||
continue;
|
||
}
|
||
let unit = cfstring_to_string(IOReportChannelGetUnitLabel(item)).unwrap_or_default();
|
||
total_uj = total_uj.saturating_add(energy_to_uj(raw, &unit));
|
||
}
|
||
total_uj
|
||
}
|
||
}
|
||
|
||
fn energy_to_uj(raw: c_long, unit: &str) -> u64 {
|
||
let r = (raw.max(0)) as f64;
|
||
let uj = match unit.trim().to_ascii_lowercase().as_str() {
|
||
"pj" => r / 1_000_000.0,
|
||
"nj" => r / 1_000.0,
|
||
"uj" | "µj" => r,
|
||
"mj" => r * 1_000.0,
|
||
"j" => r * 1_000_000.0,
|
||
_ => r / 1_000.0,
|
||
};
|
||
uj as u64
|
||
}
|
||
|
||
/// Whether the counter sums this rail, so that energy is counted once. The
|
||
/// aggregates `CPU Energy`, `GPU Energy` and `ANE Energy` (optionally
|
||
/// `DIE_<n>_`-prefixed) are summed; their summands are not. A summand is any
|
||
/// other rail whose name says CPU, GPU or ANE, and also the CPU summands
|
||
/// whose names do not: per-cluster accumulators (`PACC*`, `EACC*`, `MACC*`)
|
||
/// and CPU power managers (`*CPM*`, with their `_SRAM` rails). In one
|
||
/// absolute sample on an M5 Max, `CPU Energy` = `MCPU0` + `MCPU1` + `PCPU`,
|
||
/// and `PCPU` = `PACC_0..5` + `PCPU0..5_SRAM` + `PCPM` + `PCPM_SRAM`; on an
|
||
/// M3 Ultra, each `DIE_<n>_CPU Energy` = its `EACC_CPU` + `PACC0_CPU` +
|
||
/// `PACC1_CPU`, each the sum of its cores, their SRAM and its `_CPM` rails
|
||
/// (all to within 1-8 mJ). Summing `PACC_*` or `*_CPM` beside `CPU Energy`
|
||
/// would count that CPU energy twice.
|
||
fn counts_once(name: &str) -> bool {
|
||
let base = rail_liveness::strip_die_prefix(name);
|
||
if base.eq_ignore_ascii_case("CPU Energy")
|
||
|| base.eq_ignore_ascii_case("GPU Energy")
|
||
|| base.eq_ignore_ascii_case("ANE Energy")
|
||
{
|
||
return true;
|
||
}
|
||
let u = base.to_ascii_uppercase();
|
||
if u.contains("CPU") || u.contains("GPU") || u.contains("ANE") {
|
||
return false;
|
||
}
|
||
let cpu_summand = ["PACC", "EACC", "MACC"].iter().any(|p| u.starts_with(p)) || u.contains("CPM");
|
||
!cpu_summand
|
||
}
|
||
|
||
fn cfstring_to_string(s: CFStringRef) -> Option<String> {
|
||
if s.is_null() {
|
||
return None;
|
||
}
|
||
let mut buf = [0i8; 256];
|
||
let ok = unsafe {
|
||
CFStringGetCString(s, buf.as_mut_ptr(), buf.len() as c_long, K_CFSTRING_ENCODING_UTF8)
|
||
};
|
||
if !ok {
|
||
return None;
|
||
}
|
||
let bytes: &[u8] =
|
||
unsafe { std::slice::from_raw_parts(buf.as_ptr() as *const u8, buf.len()) };
|
||
let end = bytes.iter().position(|&b| b == 0).unwrap_or(bytes.len());
|
||
let s = std::str::from_utf8(&bytes[..end]).ok()?.trim();
|
||
if s.is_empty() { None } else { Some(s.to_string()) }
|
||
}
|
||
|
||
fn cfstring_eq_ic(s: CFStringRef, target: &str) -> bool {
|
||
cfstring_to_string(s).map(|x| x.eq_ignore_ascii_case(target)).unwrap_or(false)
|
||
}
|
||
|
||
#[cfg(test)]
|
||
mod tests {
|
||
use super::script::{self, Script};
|
||
use super::*;
|
||
use crate::rail_liveness::{CpuRail, RailKey, SampleCall};
|
||
use std::time::{Duration, Instant};
|
||
use std::cell::RefCell;
|
||
use std::rc::Rc;
|
||
use std::sync::atomic::AtomicBool;
|
||
|
||
unsafe extern "C" {
|
||
fn clock_gettime_nsec_np(clock_id: u32) -> u64;
|
||
}
|
||
|
||
/// `CLOCK_THREAD_CPUTIME_ID` in macOS `<time.h>`: the tests' own copy,
|
||
/// declared and read independently of the implementation's.
|
||
const OWN_CLOCK_THREAD_CPUTIME_ID: u32 = 16;
|
||
|
||
/// This thread's CPU time, read by the tests themselves
|
||
/// (`clock_gettime_nsec_np`), not through the implementation's clock.
|
||
fn own_thread_cpu() -> Duration {
|
||
Duration::from_nanos(unsafe { clock_gettime_nsec_np(OWN_CLOCK_THREAD_CPUTIME_ID) })
|
||
}
|
||
|
||
/// The tick the probe is specified to burn, stated here rather than
|
||
/// read from the implementation's constant.
|
||
const TICK: Duration = Duration::from_millis(25);
|
||
|
||
/// The wall time after which a tick is specified to give up, stated
|
||
/// here rather than read from the implementation's constant.
|
||
const TICK_WALL_CAP: Duration = Duration::from_millis(500);
|
||
|
||
/// How far two readings of different clocks can disagree by
|
||
/// granularity alone: both step at the 24 MHz timebase (~42 ns), so a
|
||
/// tick that ran without a break can read a few tens of ns more CPU
|
||
/// than wall time. Real CPU never exceeds wall by more.
|
||
const CLOCK_STEP: Duration = Duration::from_micros(1);
|
||
|
||
/// Real CPU work for the tests: spin this thread for `d` of wall time.
|
||
fn spin(d: Duration) {
|
||
let started = Instant::now();
|
||
let mut acc: u64 = 0;
|
||
while started.elapsed() < d {
|
||
acc = std::hint::black_box(acc.wrapping_add(1));
|
||
}
|
||
}
|
||
|
||
/// The median of a non-empty list (the mean of the middle two when even).
|
||
fn median(v: &[Duration]) -> Duration {
|
||
assert!(!v.is_empty(), "the median of nothing");
|
||
let mut v = v.to_vec();
|
||
v.sort();
|
||
let m = v.len() / 2;
|
||
if v.len().is_multiple_of(2) { (v[m - 1] + v[m]) / 2 } else { v[m] }
|
||
}
|
||
|
||
#[test]
|
||
fn energy_to_uj_scales_by_unit() {
|
||
assert_eq!(energy_to_uj(5, "mJ"), 5_000);
|
||
assert_eq!(energy_to_uj(5_000, "nJ"), 5);
|
||
assert_eq!(energy_to_uj(2, "J"), 2_000_000);
|
||
}
|
||
|
||
/// The counter sums each aggregate once and none of its summands, on
|
||
/// the real Energy Model channel names of both chips.
|
||
#[test]
|
||
fn counts_once_drops_subrails() {
|
||
// Aggregates, and domains that are no aggregate's summand.
|
||
for name in [
|
||
"CPU Energy", "DIE_0_CPU Energy", "DIE_1_CPU Energy", "GPU Energy", "ANE Energy", "AFR0", "ISP0",
|
||
"AVE0", "MSR0", "AMCC0", "DCS0", "DRAM0", "DISP0", "DISPEXT0", "FAB0", "PCIe Port 0 Energy",
|
||
"apciec0 Energy", "ISP0_1", "DRAM0_0", "AMCC0_1", "DISPEXT0_1",
|
||
] {
|
||
assert!(counts_once(name), "{name} is not summed");
|
||
}
|
||
// An M5 Max, from one absolute sample: `CPU Energy` = `MCPU0` +
|
||
// `MCPU1` + `PCPU`; `MCPUn` = its cores + their SRAM + `MCPMn` +
|
||
// `MCPMn_SRAM`; `PCPU` = `PACC_0..5` + `PCPU0..5_SRAM` + `PCPM` +
|
||
// `PCPM_SRAM`.
|
||
for name in [
|
||
"MCPU0", "MCPU1", "PCPU", "MCPU0_3", "MCPU1_5_SRAM", "PCPU2_SRAM", "PACC_0", "PACC_5", "MCPM0",
|
||
"MCPM1", "PCPM", "MCPM0_SRAM", "MCPM1_SRAM", "PCPM_SRAM", "MCPU0DTL0e",
|
||
] {
|
||
assert!(!counts_once(name), "{name}, a summand of CPU Energy on an M5 Max, is summed beside it");
|
||
}
|
||
// An M3 Ultra: `DIE_n_CPU Energy` = its `EACC_CPU` + `PACC0_CPU` +
|
||
// `PACC1_CPU`, each = its cores + their SRAM + its `_CPM` + `_CPM_SRAM`.
|
||
for name in [
|
||
"DIE_0_EACC_CPU", "DIE_1_PACC0_CPU", "DIE_0_PACC1_CPU", "DIE_0_PACC1_CPU3", "DIE_1_EACC_CPU2_SRAM",
|
||
"DIE_0_EACC_CPM", "DIE_1_PACC0_CPM", "DIE_0_PACC1_CPM_SRAM", "DIE_1_EACC_CPM_SRAM", "ECPUDTL00",
|
||
] {
|
||
assert!(!counts_once(name), "{name}, a summand of DIE_n_CPU Energy on an M3 Ultra, is summed beside it");
|
||
}
|
||
// The GPU's own rails, beside the `GPU Energy` aggregate.
|
||
for name in ["GPU0", "GPU0_0", "GPU CS0_0", "GPU SRAM0_0", "GPU CS SRAM0_0"] {
|
||
assert!(!counts_once(name), "{name} is summed beside GPU Energy");
|
||
}
|
||
}
|
||
|
||
// ── An independent oracle ────────────────────────────────────────────
|
||
|
||
/// The oracle's own CFString conversion, so a broken implementation
|
||
/// helper cannot make the oracle agree with it.
|
||
fn oracle_name(s: CFStringRef) -> String {
|
||
if s.is_null() {
|
||
return String::new();
|
||
}
|
||
let mut buf = [0u8; 512];
|
||
let ok = unsafe {
|
||
CFStringGetCString(s, buf.as_mut_ptr() as *mut i8, buf.len() as c_long, K_CFSTRING_ENCODING_UTF8)
|
||
};
|
||
if !ok {
|
||
return String::new();
|
||
}
|
||
let end = buf.iter().position(|&b| b == 0).unwrap_or(buf.len());
|
||
String::from_utf8_lossy(&buf[..end]).into_owned()
|
||
}
|
||
|
||
/// One channel as the oracle read it: its group, name and unit label,
|
||
/// and its ABSOLUTE raw integer (no delta samples).
|
||
#[derive(Debug, Clone)]
|
||
struct OracleChannel {
|
||
group: String,
|
||
name: String,
|
||
unit: String,
|
||
raw: i64,
|
||
}
|
||
|
||
/// The oracle's own subscription, over every channel (not the probe's
|
||
/// Energy Model group), opened with its own calls. It shares nothing
|
||
/// with the implementation but the raw FFI declarations.
|
||
struct OracleSub {
|
||
sub: IOReportSubscriptionRef,
|
||
chans: CFMutableDictionaryRef,
|
||
}
|
||
|
||
impl OracleSub {
|
||
/// A subscription to every channel.
|
||
fn open() -> Option<Self> {
|
||
Self::subscribe(unsafe { IOReportCopyAllChannels(0, 0) })
|
||
}
|
||
|
||
/// A subscription to the Energy Model group alone, by the oracle's
|
||
/// own calls: reads of ~7 ms instead of the 0.1-0.8 s of every
|
||
/// channel, for brackets that must be tight.
|
||
fn open_group() -> Option<Self> {
|
||
unsafe {
|
||
let group = CFStringCreateWithCString(
|
||
std::ptr::null_mut(),
|
||
c"Energy Model".as_ptr() as *const _,
|
||
K_CFSTRING_ENCODING_UTF8,
|
||
);
|
||
if group.is_null() {
|
||
return None;
|
||
}
|
||
let desired = IOReportCopyChannelsInGroup(group, std::ptr::null(), 0, 0, 0);
|
||
CFRelease(group);
|
||
Self::subscribe(desired)
|
||
}
|
||
}
|
||
|
||
/// Subscribe to `desired` (consumed; null gives `None`).
|
||
fn subscribe(desired: CFMutableDictionaryRef) -> Option<Self> {
|
||
unsafe {
|
||
if desired.is_null() {
|
||
return None;
|
||
}
|
||
let mut chans: CFMutableDictionaryRef = std::ptr::null_mut();
|
||
let sub = IOReportCreateSubscription(std::ptr::null(), desired, &mut chans, 0, std::ptr::null());
|
||
CFRelease(desired as CFTypeRef);
|
||
if sub.is_null() || chans.is_null() {
|
||
if !sub.is_null() {
|
||
CFRelease(sub);
|
||
}
|
||
if !chans.is_null() {
|
||
CFRelease(chans as CFTypeRef);
|
||
}
|
||
return None;
|
||
}
|
||
Some(Self { sub, chans })
|
||
}
|
||
}
|
||
|
||
/// One read: how many channels the sample held, in any group, and
|
||
/// every channel `keep(group, name)` accepts, in sample order.
|
||
/// `None` when IOReport returns no sample, or one with no channel
|
||
/// list.
|
||
fn read_where(&self, keep: impl Fn(&str, &str) -> bool) -> Option<(usize, Vec<OracleChannel>)> {
|
||
unsafe {
|
||
let smp = IOReportCreateSamples(self.sub, self.chans, std::ptr::null());
|
||
if smp.is_null() {
|
||
return None;
|
||
}
|
||
let key = CFStringCreateWithCString(
|
||
std::ptr::null_mut(),
|
||
c"IOReportChannels".as_ptr() as *const _,
|
||
K_CFSTRING_ENCODING_UTF8,
|
||
);
|
||
let arr = if key.is_null() {
|
||
std::ptr::null()
|
||
} else {
|
||
let arr = CFDictionaryGetValue(smp, key as *const _) as CFArrayRef;
|
||
CFRelease(key as CFTypeRef);
|
||
arr
|
||
};
|
||
let mut out = None;
|
||
if !arr.is_null() {
|
||
let count = CFArrayGetCount(arr);
|
||
let mut chans = Vec::new();
|
||
for i in 0..count {
|
||
let it = CFArrayGetValueAtIndex(arr, i);
|
||
if it.is_null() {
|
||
continue;
|
||
}
|
||
let group = oracle_name(IOReportChannelGetGroup(it));
|
||
let name = oracle_name(IOReportChannelGetChannelName(it));
|
||
if keep(&group, &name) {
|
||
chans.push(OracleChannel {
|
||
unit: oracle_name(IOReportChannelGetUnitLabel(it)),
|
||
raw: IOReportSimpleGetIntegerValue(it, 0) as i64,
|
||
group,
|
||
name,
|
||
});
|
||
}
|
||
}
|
||
out = Some((count.max(0) as usize, chans));
|
||
}
|
||
CFRelease(smp);
|
||
out
|
||
}
|
||
}
|
||
|
||
/// One read: every channel, in any group, whose name ends with `CPU
|
||
/// Energy` or `GPU Energy`, in sample order.
|
||
fn read(&self) -> Option<Vec<OracleChannel>> {
|
||
self.read_where(|_, name| name.ends_with("CPU Energy") || name.ends_with("GPU Energy")).map(|(_, c)| c)
|
||
}
|
||
|
||
/// One read's shape: (every channel, the Energy Model group's channels).
|
||
fn shape(&self) -> Option<(usize, usize)> {
|
||
self.read_where(|group, _| group == "Energy Model").map(|(all, group)| (all, group.len()))
|
||
}
|
||
|
||
/// One read of the Energy Model group's CPU and GPU energy rails (the
|
||
/// oracle's own name rule), keyed by (name, ordinal among the same
|
||
/// name), in sample order.
|
||
fn energy_rails(&self) -> Vec<(RailKey, i64)> {
|
||
let (_, chans) = self
|
||
.read_where(|group, name| {
|
||
group == "Energy Model" && (name.ends_with("CPU Energy") || name.ends_with("GPU Energy"))
|
||
})
|
||
.expect("an oracle read");
|
||
oracle_keyed(chans.into_iter().map(|c| (c.name, c.raw)))
|
||
}
|
||
}
|
||
|
||
impl Drop for OracleSub {
|
||
fn drop(&mut self) {
|
||
unsafe {
|
||
CFRelease(self.sub);
|
||
CFRelease(self.chans as CFTypeRef);
|
||
}
|
||
}
|
||
}
|
||
|
||
/// (name, value) pairs keyed by (name, ordinal among the same name), in
|
||
/// order: the oracle's own keying.
|
||
fn oracle_keyed<V>(items: impl IntoIterator<Item = (String, V)>) -> Vec<(RailKey, V)> {
|
||
let mut out: Vec<(RailKey, V)> = Vec::new();
|
||
for (name, v) in items {
|
||
let ordinal = out.iter().filter(|((n, _), _)| *n == name).count() as u32;
|
||
out.push(((name, ordinal), v));
|
||
}
|
||
out
|
||
}
|
||
|
||
/// What the oracle saw on this host.
|
||
#[derive(Debug, PartialEq)]
|
||
enum Oracle {
|
||
NoIoReport,
|
||
NoCpuRail,
|
||
CpuRailMoved(bool),
|
||
}
|
||
|
||
#[derive(Debug)]
|
||
struct OracleRun {
|
||
verdict: Oracle,
|
||
reads: u32,
|
||
/// Pairs of consecutive reads compared since the last baseline:
|
||
/// what the per-rail change counts are out of.
|
||
compared: u32,
|
||
took: Duration,
|
||
cpu_changes: Vec<u32>,
|
||
gpu_changes: Vec<u32>,
|
||
}
|
||
|
||
/// The fewest changes at which one rail counts as moving, and so the
|
||
/// fewest pairs of consecutive reads the oracle compares (six reads)
|
||
/// before its verdict can say anything: it reads past its window until
|
||
/// it has this many, so a live rail, changing at every read, can meet
|
||
/// its bar. Its own literal, not the implementation's constant.
|
||
const ORACLE_MIN_CHANGES: u32 = 5;
|
||
|
||
/// However slow its reads, the oracle stops after this many windows.
|
||
const ORACLE_WINDOW_CAP: u32 = 5;
|
||
|
||
/// The oracle's bar for moving CPU rails, over `compared` pairs of
|
||
/// consecutive reads: EVERY CPU rail it found (each die's, on a
|
||
/// multi-die part) changed at least [`ORACLE_MIN_CHANGES`] times, at no
|
||
/// fewer than 80% of them. A live rail changes at nearly every read. A
|
||
/// refresh — one step, or two 64 ms apart, or a few in a burst —
|
||
/// changes at a handful of reads in a window of 0.1-0.8 s reads, so it
|
||
/// cannot vouch for a Live verdict it happened to coincide with. Every
|
||
/// rail, because the counter sums every die: one die moving beside one
|
||
/// that holds cannot vouch for a Live verdict either.
|
||
fn oracle_moved(cpu_changes: &[u32], compared: u32) -> bool {
|
||
!cpu_changes.is_empty()
|
||
&& cpu_changes.iter().all(|&n| n >= ORACLE_MIN_CHANGES && u64::from(n) * 5 >= u64::from(compared) * 4)
|
||
}
|
||
|
||
/// What the oracle has counted, read by read: per rail, at how many
|
||
/// reads its value changed since the read before, and out of how many
|
||
/// pairs of reads compared since the last baseline. Pure, so the bar
|
||
/// runs under test on scripted reads.
|
||
#[derive(Debug, Default)]
|
||
struct OracleTally {
|
||
prev: Option<(Vec<OracleChannel>, Vec<OracleChannel>)>,
|
||
saw_cpu_rail: bool,
|
||
compared: u32,
|
||
cpu_changes: Vec<u32>,
|
||
gpu_changes: Vec<u32>,
|
||
}
|
||
|
||
impl OracleTally {
|
||
/// Fold in one read: every channel whose name ends with `CPU
|
||
/// Energy` or `GPU Energy`, in sample order.
|
||
fn fold(&mut self, chans: Vec<OracleChannel>) {
|
||
let names = |v: &[OracleChannel]| v.iter().map(|c| c.name.clone()).collect::<Vec<_>>();
|
||
let (cpu, gpu): (Vec<_>, Vec<_>) = chans.into_iter().partition(|c| c.name.ends_with("CPU Energy"));
|
||
self.saw_cpu_rail |= !cpu.is_empty();
|
||
match &self.prev {
|
||
Some((pc, pg)) if names(pc) == names(&cpu) && names(pg) == names(&gpu) => {
|
||
self.compared += 1;
|
||
for (n, (now, before)) in self.cpu_changes.iter_mut().zip(cpu.iter().zip(pc)) {
|
||
*n += u32::from(now.raw != before.raw);
|
||
}
|
||
for (n, (now, before)) in self.gpu_changes.iter_mut().zip(gpu.iter().zip(pg)) {
|
||
*n += u32::from(now.raw != before.raw);
|
||
}
|
||
}
|
||
// The first read, or rails that differ from the last read's: a new baseline.
|
||
_ => {
|
||
self.compared = 0;
|
||
self.cpu_changes = vec![0; cpu.len()];
|
||
self.gpu_changes = vec![0; gpu.len()];
|
||
}
|
||
}
|
||
self.prev = Some((cpu, gpu));
|
||
}
|
||
|
||
fn verdict(&self) -> Oracle {
|
||
if self.saw_cpu_rail {
|
||
Oracle::CpuRailMoved(oracle_moved(&self.cpu_changes, self.compared))
|
||
} else {
|
||
Oracle::NoCpuRail
|
||
}
|
||
}
|
||
}
|
||
|
||
impl OracleRun {
|
||
/// The oracle's evidence is only as good as its reads. Fewer than
|
||
/// five compared pairs cannot show a rail changing five times, so
|
||
/// they say nothing about whether the CPU rail is frozen: that is a
|
||
/// test FAILURE, never a Frozen verdict. With no subscription there
|
||
/// are no reads to have.
|
||
fn assert_evidence(&self) {
|
||
if self.verdict == Oracle::NoIoReport {
|
||
return;
|
||
}
|
||
assert!(
|
||
self.compared >= ORACLE_MIN_CHANGES,
|
||
"the oracle compared only {} pairs of consecutive reads ({} reads) in {:?}, though it keeps \
|
||
reading past its window until it has {ORACLE_MIN_CHANGES}, for up to {ORACLE_WINDOW_CAP} \
|
||
windows: too few for any rail to change {ORACLE_MIN_CHANGES} times, so there is no evidence \
|
||
about the CPU rail either way",
|
||
self.compared,
|
||
self.reads,
|
||
self.took
|
||
);
|
||
}
|
||
}
|
||
|
||
/// The oracle's view of the CPU rail, over at least `window` while a
|
||
/// HELPER thread burns CPU the whole time (so the window is under load
|
||
/// even while an all-channel read blocks this thread).
|
||
///
|
||
/// It reads its own all-channel subscription over and over, matches
|
||
/// names with `ends_with("CPU Energy")` in any group (not the
|
||
/// implementation's group filter and die-prefix stripping), and counts
|
||
/// per rail at how many reads the value changed since the read before
|
||
/// ([`OracleTally`]). `CpuRailMoved(true)` only by [`oracle_moved`]:
|
||
/// every CPU rail changed at 80% or more of the compared reads, and at
|
||
/// least five times. It reads until the window has passed AND it has compared
|
||
/// [`ORACLE_MIN_CHANGES`] pairs of reads, for at most
|
||
/// [`ORACLE_WINDOW_CAP`] windows, so slow reads on a loaded host
|
||
/// lengthen the window rather than thin the evidence.
|
||
fn oracle_cpu_rail(window: Duration) -> OracleRun {
|
||
let Some(sub) = OracleSub::open() else {
|
||
return OracleRun {
|
||
verdict: Oracle::NoIoReport,
|
||
reads: 0,
|
||
compared: 0,
|
||
took: Duration::ZERO,
|
||
cpu_changes: Vec::new(),
|
||
gpu_changes: Vec::new(),
|
||
};
|
||
};
|
||
let stop = AtomicBool::new(false);
|
||
std::thread::scope(|s| {
|
||
s.spawn(|| {
|
||
let mut acc: u64 = 0;
|
||
while !stop.load(Ordering::Relaxed) {
|
||
acc = std::hint::black_box(acc.wrapping_add(1));
|
||
}
|
||
});
|
||
let started = Instant::now();
|
||
let (mut reads, mut tally) = (0u32, OracleTally::default());
|
||
while (started.elapsed() < window || tally.compared < ORACLE_MIN_CHANGES)
|
||
&& started.elapsed() < window * ORACLE_WINDOW_CAP
|
||
{
|
||
let Some(chans) = sub.read() else {
|
||
continue;
|
||
};
|
||
reads += 1;
|
||
tally.fold(chans);
|
||
}
|
||
stop.store(true, Ordering::Relaxed);
|
||
OracleRun {
|
||
verdict: tally.verdict(),
|
||
reads,
|
||
compared: tally.compared,
|
||
took: started.elapsed(),
|
||
cpu_changes: tally.cpu_changes,
|
||
gpu_changes: tally.gpu_changes,
|
||
}
|
||
})
|
||
}
|
||
|
||
/// A scripted oracle read: `CPU Energy` at `cpu` and a `GPU Energy`
|
||
/// that moves at every read.
|
||
fn oracle_read(i: usize, cpu: i64) -> Vec<OracleChannel> {
|
||
let chan = |name: &str, raw: i64| OracleChannel {
|
||
group: "Energy Model".to_string(),
|
||
name: name.to_string(),
|
||
unit: "mJ".to_string(),
|
||
raw,
|
||
};
|
||
vec![chan("CPU Energy", cpu), chan("GPU Energy", 1_000 + 7 * i as i64)]
|
||
}
|
||
|
||
/// The oracle cannot vouch for a Live verdict with a refresh. Over a
|
||
/// window of scripted reads, the observed two-step refresh (+61,764,485
|
||
/// then +3,435), a single step, and a five-step burst are no movement;
|
||
/// a rail changing at every read, or at four reads in five, is; one
|
||
/// changing at three reads in four is not, nor one that changed at
|
||
/// every one of only four compared pairs. The old bar (two changes)
|
||
/// called the two-step refresh movement.
|
||
#[test]
|
||
fn the_oracle_does_not_count_a_refresh_as_movement() {
|
||
let tally = |reads: usize, cpu: &dyn Fn(usize) -> i64| {
|
||
let mut t = OracleTally::default();
|
||
for i in 0..reads {
|
||
t.fold(oracle_read(i, cpu(i)));
|
||
}
|
||
t
|
||
};
|
||
let a = 2_643_843_378;
|
||
let two_step = |i: usize| if i < 4 { a } else if i < 5 { a + 61_764_485 } else { a + 61_764_485 + 3_435 };
|
||
let t = tally(12, &two_step);
|
||
assert_eq!(
|
||
(t.verdict(), t.compared, t.cpu_changes.clone(), t.gpu_changes.clone()),
|
||
(Oracle::CpuRailMoved(false), 11, vec![2], vec![11]),
|
||
"the oracle counted the two-step refresh as a moving CPU rail"
|
||
);
|
||
let one_step = |i: usize| if i < 4 { a } else { a + 61_764_485 };
|
||
assert_eq!(tally(12, &one_step).verdict(), Oracle::CpuRailMoved(false));
|
||
let burst = |i: usize| a + 100 * (i.clamp(3, 8) as i64 - 3);
|
||
let t = tally(20, &burst);
|
||
assert_eq!((t.verdict(), t.cpu_changes.clone()), (Oracle::CpuRailMoved(false), vec![5]), "a five-step burst");
|
||
|
||
let live = |i: usize| a + 13 * i as i64;
|
||
assert_eq!(tally(6, &live).verdict(), Oracle::CpuRailMoved(true), "5 changes in 5 compared pairs");
|
||
assert_eq!(tally(5, &live).verdict(), Oracle::CpuRailMoved(false), "4 changes in 4 compared pairs");
|
||
let four_in_five = |i: usize| a + (1..=i).filter(|k| k % 5 != 0).count() as i64;
|
||
assert_eq!(tally(21, &four_in_five).verdict(), Oracle::CpuRailMoved(true), "16 changes in 20");
|
||
let three_in_four = |i: usize| a + (1..=i).filter(|k| k % 4 != 0).count() as i64;
|
||
assert_eq!(tally(21, &three_in_four).verdict(), Oracle::CpuRailMoved(false), "15 changes in 20");
|
||
assert!(!oracle_moved(&[2], 11) && !oracle_moved(&[3, 3], 5) && !oracle_moved(&[], 11));
|
||
// Two dies: both must have moved.
|
||
assert!(oracle_moved(&[11, 9], 11), "two moving dies");
|
||
assert!(!oracle_moved(&[0, 11], 11) && !oracle_moved(&[11, 2], 11), "one die moving beside one that holds");
|
||
}
|
||
|
||
// ── The production status, on scripted IOReport leaves ───────────────
|
||
|
||
/// A sample the way the FFI returns one of the Energy Model group:
|
||
/// every channel in the group.
|
||
fn group_sample(values: &[(&str, i64)]) -> Sample {
|
||
Sample {
|
||
channels: values.len(),
|
||
group_only: true,
|
||
values: values.iter().map(|(n, raw)| (n.to_string(), *raw)).collect(),
|
||
}
|
||
}
|
||
|
||
/// One production tick as the tests' own clocks saw it.
|
||
#[derive(Debug, Clone, Copy)]
|
||
struct TickSeen {
|
||
cpu: Duration,
|
||
wall: Duration,
|
||
}
|
||
|
||
/// What a script saw of the production ticks: when it last returned a
|
||
/// sample (wall, then this thread's CPU), and every tick since the first.
|
||
#[derive(Default)]
|
||
struct TickMarks {
|
||
returned: Option<(Instant, Duration)>,
|
||
ticks: Vec<TickSeen>,
|
||
}
|
||
|
||
/// The production status — the counted, uncached [`host_status`] —
|
||
/// with a script standing in for IOReport's two FFI leaves: the group
|
||
/// subscription opens or not, and `sample(i)` is its i-th sample.
|
||
/// Returns the status, how many samples the script served, and every
|
||
/// production tick as the tests' own clocks saw it: from the moment the
|
||
/// script returned one sample to the moment it was asked for the next,
|
||
/// with the CPU reading nested inside the wall reading.
|
||
fn scripted_host_status(
|
||
opens: bool,
|
||
sample: impl Fn(usize) -> Option<Sample> + 'static,
|
||
) -> (IoReportStatus, usize, Vec<TickSeen>) {
|
||
let seen: Rc<RefCell<TickMarks>> = Rc::default();
|
||
let marks = Rc::clone(&seen);
|
||
let script = Script {
|
||
opens,
|
||
samples: Box::new(move |i| {
|
||
let (cpu, wall) = (own_thread_cpu(), Instant::now());
|
||
let mut m = marks.borrow_mut();
|
||
if let Some((wall0, cpu0)) = m.returned {
|
||
m.ticks.push(TickSeen { cpu: cpu.saturating_sub(cpu0), wall: wall.saturating_duration_since(wall0) });
|
||
}
|
||
assert!(i < 10_000, "the probe took {i} samples in one budget: its tick does not burn CPU");
|
||
let s = sample(i);
|
||
m.returned = Some((Instant::now(), own_thread_cpu()));
|
||
s
|
||
}),
|
||
};
|
||
let (status, served) = script::with(script, host_status);
|
||
let ticks = seen.borrow().ticks.clone();
|
||
(status, served, ticks)
|
||
}
|
||
|
||
/// The production status itself: [`host_status`] — the real opener's
|
||
/// sampler closure, [`production_probe`] with the real tick and clocks,
|
||
/// the rule and the mapping — with a script standing in for IOReport's
|
||
/// two FFI leaves. No subscription is `Unavailable`, with no sample
|
||
/// taken; a single batch refresh is `CpuRailFrozen`; the observed
|
||
/// refresh published in two steps is `CpuRailFrozen`; a rail advancing
|
||
/// at every sample is `Live`, once five intervals and 300 ms of the
|
||
/// real wall clock have passed and no later; one step on each of two
|
||
/// dies, in different intervals, is `CpuRailFrozen`, and so is one die
|
||
/// advancing at every sample beside one that holds. These hosts
|
||
/// produce none of the first four on demand, so no hardware test can
|
||
/// reach them. Every scripted step falls in the first five intervals,
|
||
/// which the probe always observes however loaded the host: it takes
|
||
/// at least six samples unless the rail proves live first.
|
||
#[test]
|
||
fn host_status_is_the_rule_on_the_production_composition() {
|
||
let (status, served, ticks) = scripted_host_status(false, |_| panic!("sampled without a subscription"));
|
||
assert_eq!(
|
||
(status, served, ticks.len()),
|
||
(IoReportStatus::Unavailable, 0, 0),
|
||
"no group subscription must be Unavailable, with no sample and no tick"
|
||
);
|
||
|
||
let refresh =
|
||
|i: usize| [("CPU Energy", if i < 1 { 480_083_160 } else { 541_893_638 }), ("GPU Energy", 1_000 + 7 * i as i64)];
|
||
let started = Instant::now();
|
||
let (status, served, _) = scripted_host_status(true, move |i| Some(group_sample(&refresh(i))));
|
||
let took = started.elapsed();
|
||
assert_eq!(status, IoReportStatus::CpuRailFrozen, "a single batch refresh ({served} samples) must be frozen");
|
||
// A frozen verdict takes the whole budget, 1 s of wall time: the probe
|
||
// samples until it has passed. Load can only lengthen this.
|
||
assert!(took >= Duration::from_secs(1), "the production probe gave a frozen verdict after {took:?}");
|
||
|
||
// The M5 Max's two-step refresh: +61,764,485 then +3,435, in the
|
||
// third and fourth intervals. The old rule's two advances.
|
||
let two_step = |i: usize| {
|
||
let a = 2_643_843_378;
|
||
let cpu = if i < 3 { a } else if i < 4 { a + 61_764_485 } else { a + 61_764_485 + 3_435 };
|
||
[("CPU Energy", cpu), ("MCPU0", cpu / 4), ("PCPU", cpu / 2), ("GPU Energy", 1_000 + 7 * i as i64)]
|
||
};
|
||
let (status, served, _) = scripted_host_status(true, move |i| Some(group_sample(&two_step(i))));
|
||
assert_eq!(status, IoReportStatus::CpuRailFrozen, "a refresh in two steps ({served} samples) must be frozen");
|
||
|
||
// A rail advancing at every sample. Each production tick burns 25 ms
|
||
// of this thread's CPU, so at least 25 ms of wall time: 300 ms have
|
||
// passed by the 12th interval at the latest (13 allowed, for clock
|
||
// granularity), and never before the 5th.
|
||
let live = |i: usize| [("GPU Energy", 5), ("CPU Energy", 1_000 + 13 * i as i64)];
|
||
let started = Instant::now();
|
||
let (status, served, _) = scripted_host_status(true, move |i| Some(group_sample(&live(i))));
|
||
let took = started.elapsed();
|
||
assert_eq!(status, IoReportStatus::Live, "a rail advancing at every one of {served} samples must be live");
|
||
assert!(
|
||
(6..=14).contains(&served) && took >= Duration::from_millis(300),
|
||
"a sustained advance was declared live after {served} samples in {took:?}: the rule needs five \
|
||
intervals and 300 ms, and the probe stops as soon as it holds"
|
||
);
|
||
|
||
let dies = |i: usize| {
|
||
[
|
||
("DIE_0_CPU Energy", if i >= 1 { 9_500_000 } else { 9_494_336 }),
|
||
("DIE_1_CPU Energy", if i >= 2 { 10_300_000 } else { 10_212_656 }),
|
||
]
|
||
};
|
||
let (status, served, _) = scripted_host_status(true, move |i| Some(group_sample(&dies(i))));
|
||
assert_eq!(status, IoReportStatus::CpuRailFrozen, "one step on each die ({served} samples) must be frozen");
|
||
|
||
// DIE_0 advancing at every sample, DIE_1 held: the counter would
|
||
// sum a live die and a frozen one, so the whole is frozen.
|
||
let one_live_die = |i: usize| [("DIE_0_CPU Energy", 9_494_336 + 1_307 * i as i64), ("DIE_1_CPU Energy", 10_212_656)];
|
||
let (status, served, _) = scripted_host_status(true, move |i| Some(group_sample(&one_live_die(i))));
|
||
assert_eq!(status, IoReportStatus::CpuRailFrozen, "one live die beside a frozen one ({served} samples) must be frozen");
|
||
}
|
||
|
||
/// Each production tick, in the production composition, burns 25 ms of
|
||
/// this thread's CPU by the tests' own clock, or ran into its 500 ms
|
||
/// wall cap; and none shows more CPU than wall time. Between one
|
||
/// scripted sample and the next the production code runs its tick and
|
||
/// only bookkeeping besides, so the script's own clock readings
|
||
/// bracket each tick.
|
||
#[test]
|
||
fn each_production_tick_burns_its_cpu_by_the_tests_own_clock() {
|
||
let (status, served, ticks) =
|
||
scripted_host_status(true, |i| Some(group_sample(&[("CPU Energy", 42), ("GPU Energy", i as i64)])));
|
||
let cpus: Vec<Duration> = ticks.iter().map(|t| t.cpu).collect();
|
||
let walls: Vec<Duration> = ticks.iter().map(|t| t.wall).collect();
|
||
eprintln!(
|
||
"ioreport production ticks (the tests' own clocks): {} ticks between {served} scripted samples; \
|
||
thread CPU min {:?} median {:?} max {:?}; wall min {:?} max {:?}",
|
||
ticks.len(),
|
||
cpus.iter().min(),
|
||
(!cpus.is_empty()).then(|| median(&cpus)),
|
||
cpus.iter().max(),
|
||
walls.iter().min(),
|
||
walls.iter().max()
|
||
);
|
||
assert_eq!(status, IoReportStatus::CpuRailFrozen);
|
||
assert!(ticks.len() >= 5 && ticks.len() == served - 1, "{} ticks for {served} samples", ticks.len());
|
||
for (i, t) in ticks.iter().enumerate() {
|
||
assert!(
|
||
t.cpu >= TICK || t.wall >= TICK_WALL_CAP,
|
||
"production tick {i} burned {:?} of this thread's CPU in {:?} of wall time (every tick: {ticks:?}): \
|
||
a tick burns 25 ms of the thread's CPU, or gives up after 500 ms of wall time",
|
||
t.cpu,
|
||
t.wall
|
||
);
|
||
assert!(
|
||
t.cpu <= t.wall + CLOCK_STEP,
|
||
"production tick {i}: {:?} of this thread's CPU in {:?} of wall time",
|
||
t.cpu,
|
||
t.wall
|
||
);
|
||
}
|
||
}
|
||
|
||
/// The production tick burns thread CPU, not wall time. With every core
|
||
/// oversubscribed by busy threads, a tick that spun for 25 ms of WALL
|
||
/// time would get only part of that as CPU; the production tick keeps
|
||
/// spinning until this thread's own CPU clock has advanced 25 ms (or its
|
||
/// 500 ms wall cap). Pinned per tick, by the tests' own clocks, in the
|
||
/// production composition, over a frozen verdict's whole budget, which
|
||
/// always holds five ticks at least; and the contention must have been
|
||
/// real: at least one tick shared its core. The probe starts only once
|
||
/// every busy thread is running.
|
||
#[test]
|
||
fn the_production_tick_burns_cpu_not_wall_time_on_a_shared_core() {
|
||
let spinners = 2 * std::thread::available_parallelism().map_or(8, |n| n.get());
|
||
let (stop, running) = (AtomicBool::new(false), AtomicUsize::new(0));
|
||
let (status, served, ticks) = std::thread::scope(|s| {
|
||
// Stops the spinners however the scope's body ends.
|
||
struct Stop<'a>(&'a AtomicBool);
|
||
impl Drop for Stop<'_> {
|
||
fn drop(&mut self) {
|
||
self.0.store(true, Ordering::Relaxed);
|
||
}
|
||
}
|
||
let _stop = Stop(&stop);
|
||
for _ in 0..spinners {
|
||
s.spawn(|| {
|
||
running.fetch_add(1, Ordering::SeqCst);
|
||
let mut acc: u64 = 0;
|
||
while !stop.load(Ordering::Relaxed) {
|
||
acc = std::hint::black_box(acc.wrapping_add(1));
|
||
}
|
||
});
|
||
}
|
||
while running.load(Ordering::SeqCst) < spinners {
|
||
std::thread::yield_now();
|
||
}
|
||
scripted_host_status(true, |i| Some(group_sample(&[("CPU Energy", 42), ("GPU Energy", i as i64)])))
|
||
});
|
||
let shared = ticks.iter().filter(|t| t.wall > t.cpu + Duration::from_millis(5)).count();
|
||
eprintln!(
|
||
"ioreport production ticks on a shared core ({spinners} busy threads): status={status:?} after {served} \
|
||
samples; {shared} of {} ticks shared their core; every tick (cpu, wall): {:?}",
|
||
ticks.len(),
|
||
ticks.iter().map(|t| (t.cpu, t.wall)).collect::<Vec<_>>()
|
||
);
|
||
assert_eq!(status, IoReportStatus::CpuRailFrozen);
|
||
assert!(ticks.len() >= 5 && ticks.len() == served - 1, "{} ticks for {served} samples", ticks.len());
|
||
for (i, t) in ticks.iter().enumerate() {
|
||
assert!(
|
||
t.cpu >= TICK || t.wall >= TICK_WALL_CAP,
|
||
"production tick {i} burned {:?} of this thread's CPU in {:?} of wall time on a shared core: \
|
||
the tick must spin until the thread's CPU clock has advanced 25 ms, not for 25 ms of wall time",
|
||
t.cpu,
|
||
t.wall
|
||
);
|
||
assert!(
|
||
t.cpu <= t.wall + CLOCK_STEP,
|
||
"production tick {i}: {:?} of this thread's CPU in {:?} of wall time",
|
||
t.cpu,
|
||
t.wall
|
||
);
|
||
}
|
||
assert!(
|
||
shared >= 1,
|
||
"no tick shared its core with the {spinners} busy threads ({ticks:?}), so this run could not tell a \
|
||
tick on the thread's CPU clock from one on the wall clock"
|
||
);
|
||
}
|
||
|
||
/// The implementation's thread clock is this thread's CPU time, checked
|
||
/// against the tests' own `CLOCK_THREAD_CPUTIME_ID`: it keeps pace with
|
||
/// the tests' clock while this thread burns CPU; it does not advance
|
||
/// across a 100 ms sleep (so it is no wall clock); and it does not
|
||
/// advance while another thread burns 100 ms of CPU and this one waits
|
||
/// (so it is not the process's CPU clock).
|
||
#[test]
|
||
fn the_probe_thread_clock_is_this_threads_cpu_time() {
|
||
let ms = Duration::from_millis;
|
||
let (own0, p0) = (own_thread_cpu(), thread_cpu_time());
|
||
while own_thread_cpu() - own0 < ms(20) {
|
||
std::hint::black_box(0u64);
|
||
}
|
||
let (p1, own1) = (thread_cpu_time(), own_thread_cpu());
|
||
let (burned, own) = (p1.saturating_sub(p0), own1 - own0);
|
||
|
||
let p0 = thread_cpu_time();
|
||
std::thread::sleep(ms(100));
|
||
let slept = thread_cpu_time().saturating_sub(p0);
|
||
|
||
let p0 = thread_cpu_time();
|
||
let helper = std::thread::scope(|s| {
|
||
s.spawn(|| {
|
||
let c0 = own_thread_cpu();
|
||
let mut acc: u64 = 0;
|
||
while own_thread_cpu() - c0 < ms(100) {
|
||
acc = std::hint::black_box(acc.wrapping_add(1));
|
||
}
|
||
own_thread_cpu() - c0
|
||
})
|
||
.join()
|
||
.expect("the helper thread panicked")
|
||
});
|
||
let waited = thread_cpu_time().saturating_sub(p0);
|
||
eprintln!(
|
||
"ioreport thread clock: {burned:?} while this thread burned {own:?} (tests' own clock); {slept:?} across \
|
||
a 100 ms sleep; {waited:?} while a helper burned {helper:?} and this thread waited"
|
||
);
|
||
assert!(
|
||
burned <= own && own - burned < ms(1),
|
||
"the implementation's thread clock advanced {burned:?} while the tests' own thread clock advanced {own:?}"
|
||
);
|
||
assert!(
|
||
slept < ms(5),
|
||
"the implementation's thread clock advanced {slept:?} across a 100 ms sleep: it reads wall time, not \
|
||
this thread's CPU"
|
||
);
|
||
assert!(helper >= ms(100), "the helper burned only {helper:?}");
|
||
assert!(
|
||
waited < ms(5),
|
||
"the implementation's thread clock advanced {waited:?} while another thread burned {helper:?} of CPU and \
|
||
this one waited: it reads the process's CPU, not this thread's"
|
||
);
|
||
}
|
||
|
||
/// A declined status declines before anything opens: given
|
||
/// `Unavailable`, `CpuRailAbsent` or `CpuRailFrozen`, the constructor
|
||
/// returns `Err(NoCounter)` without trying to open the counter's
|
||
/// subscription, and so does selection, whatever IOReport does on this
|
||
/// host. Given `Live`, the constructor does try to open it — the count
|
||
/// is live.
|
||
#[test]
|
||
fn a_declined_status_declines_before_anything_opens() {
|
||
use IoReportStatus as S;
|
||
for status in [S::Unavailable, S::CpuRailAbsent, S::CpuRailFrozen] {
|
||
let before = script::counter_opens();
|
||
let given = MacosIoReportCounter::try_detect_given(status);
|
||
assert!(matches!(given, Err(Error::NoCounter)), "try_detect_given({status:?}) did not decline with NoCounter");
|
||
let picked = crate::detect_with(|| CounterStatus::Unavailable, || status).name();
|
||
assert_ne!(picked, "ioreport", "detect_with({status:?}) selected the IOReport counter");
|
||
let (built, picked) =
|
||
script::with_status(status, || (MacosIoReportCounter::try_detect().is_ok(), crate::detect().name()));
|
||
assert!(!built && picked != "ioreport", "try_detect()/detect() under {status:?}: built={built} picked={picked}");
|
||
assert_eq!(
|
||
script::counter_opens(),
|
||
before,
|
||
"under {status:?} the constructor or selection tried to open the counter's subscription before declining"
|
||
);
|
||
}
|
||
let before = script::counter_opens();
|
||
let _ = MacosIoReportCounter::try_detect_given(S::Live);
|
||
assert_eq!(script::counter_opens(), before + 1, "given Live, the constructor did not try to open its subscription");
|
||
}
|
||
|
||
/// Every constructor and selection path gates on the pure
|
||
/// `admits_counter`: given each declined status, `try_detect_given`
|
||
/// declines, and `detect_with` falls through to a counter that is not
|
||
/// IOReport; given `Live`, both build the IOReport counter wherever the
|
||
/// all-channel subscription opens. And the real `try_detect()` and
|
||
/// `detect()`, run with each status installed for `ioreport_status()`
|
||
/// on this thread, do the same; with none installed they agree with
|
||
/// the cached status.
|
||
#[test]
|
||
fn only_a_live_status_constructs_or_selects_the_counter() {
|
||
use IoReportStatus as S;
|
||
let opens = MacosIoReportCounter::open().is_some();
|
||
for status in [S::Unavailable, S::CpuRailAbsent, S::CpuRailFrozen, S::Live] {
|
||
let admitted = status == S::Live && opens;
|
||
let given = MacosIoReportCounter::try_detect_given(status);
|
||
assert_eq!(given.is_ok(), admitted, "try_detect_given({status:?}), the subscription opening={opens}");
|
||
if let Err(e) = given {
|
||
assert!(matches!(e, Error::NoCounter), "try_detect_given({status:?}) declined with {e:?}");
|
||
}
|
||
let picked = crate::detect_with(|| CounterStatus::Unavailable, || status).name();
|
||
assert_eq!(picked == "ioreport", admitted, "detect_with({status:?}) selected {picked}");
|
||
let (built, picked) = script::with_status(status, || {
|
||
assert_eq!(crate::ioreport_status(), status, "the installed status");
|
||
(MacosIoReportCounter::try_detect().is_ok(), crate::detect().name())
|
||
});
|
||
assert_eq!(built, admitted, "try_detect() with the status {status:?}, the subscription opening={opens}");
|
||
assert_eq!(picked == "ioreport", admitted, "detect() with the status {status:?} selected {picked}");
|
||
}
|
||
let status = crate::ioreport_status();
|
||
assert_eq!(MacosIoReportCounter::try_detect().is_ok(), status == S::Live && opens, "status={status:?}");
|
||
assert_eq!(crate::detect().name() == "ioreport", status == S::Live && opens, "status={status:?}");
|
||
}
|
||
|
||
// ── Against the hardware ─────────────────────────────────────────────
|
||
|
||
/// The property that matters: the IOReport counter is **never**
|
||
/// constructed, and never picked by [`crate::detect`], when a CPU rail
|
||
/// it sums is not live — then it would report GPU-only energy, zero, a
|
||
/// batch refresh, or one die's energy as the energy of CPU work, and
|
||
/// consumers would tag it Measured. Asserts the meter's decision against what the rail actually
|
||
/// did; every branch asserts, so there is no way to pass without
|
||
/// checking anything, and an oracle with too few reads is a failure.
|
||
///
|
||
/// The oracle gets at least three times the probe's budget, so a rail too
|
||
/// slow for the probe but moving within the oracle's window is the only
|
||
/// case where declining is accepted — and that fails safe: estimated,
|
||
/// never over-claimed. A batch refresh in both windows does not rescue
|
||
/// a probe that called it live, even one published in two or more
|
||
/// steps: the oracle counts a rail as moving only if it changed at 80%
|
||
/// or more of its reads, at least five times, and the CPU as moving only
|
||
/// if every CPU rail it found did.
|
||
#[test]
|
||
fn the_ioreport_counter_is_never_selected_on_a_frozen_cpu_rail() {
|
||
let status = crate::ioreport_status();
|
||
let constructible = MacosIoReportCounter::try_detect().is_ok();
|
||
let selected = crate::detect().name() == "ioreport";
|
||
let run = oracle_cpu_rail(crate::CPU_RAIL_LIVENESS_BUDGET * 3);
|
||
// One line, so a CI log says which branch below was actually asserted.
|
||
eprintln!(
|
||
"ioreport liveness: status={status:?} oracle={:?} selected={selected} \
|
||
(oracle: {} reads over {:?} with a helper thread burning CPU; \
|
||
changes per CPU rail {:?}, per GPU rail {:?}, of {} compared pairs)",
|
||
run.verdict, run.reads, run.took, run.cpu_changes, run.gpu_changes, run.compared
|
||
);
|
||
run.assert_evidence();
|
||
match run.verdict {
|
||
Oracle::NoIoReport => {
|
||
assert!(!selected, "IOReport selected (status={status:?}) but no subscription opens");
|
||
assert_eq!(status, IoReportStatus::Unavailable);
|
||
}
|
||
Oracle::NoCpuRail => {
|
||
assert!(!selected, "IOReport selected (status={status:?}) but it has no CPU rail");
|
||
assert!(
|
||
matches!(status, IoReportStatus::CpuRailAbsent | IoReportStatus::Unavailable),
|
||
"the oracle found no CPU rail, the status is {status:?}"
|
||
);
|
||
}
|
||
Oracle::CpuRailMoved(false) => {
|
||
assert!(
|
||
!selected,
|
||
"IOReport selected (status={status:?}) although its CPU rail did not keep moving \
|
||
under {:?} of CPU burn — it would report GPU-only energy, zero, or a batch \
|
||
refresh as the energy of CPU work",
|
||
run.took
|
||
);
|
||
assert_eq!(
|
||
status,
|
||
IoReportStatus::CpuRailFrozen,
|
||
"the oracle did not see the CPU rail keep moving under {:?} of CPU burn",
|
||
run.took
|
||
);
|
||
}
|
||
Oracle::CpuRailMoved(true) => {
|
||
// A live rail. The probe had a third of the oracle's time: if it
|
||
// saw the rail keep advancing the counter must be selected; if
|
||
// not, declining is the safe side.
|
||
assert!(
|
||
matches!(status, IoReportStatus::Live | IoReportStatus::CpuRailFrozen),
|
||
"the oracle saw every CPU rail keep moving but the status is {status:?}"
|
||
);
|
||
assert_eq!(selected, status == IoReportStatus::Live, "status={status:?}");
|
||
}
|
||
}
|
||
// On macOS IOReport is the first rung, so detect() picks it exactly when
|
||
// try_detect() constructs it, and that is exactly when the status is Live.
|
||
assert_eq!(selected, constructible, "detect() and try_detect() disagree");
|
||
assert_eq!(constructible, status == IoReportStatus::Live);
|
||
}
|
||
|
||
/// The dangerous direction on its own, against the production
|
||
/// composition run fresh (uncounted, uncached) beside the cached status:
|
||
/// it never reports Live for a rail the oracle did not see keep moving,
|
||
/// and on every host both agree with the oracle about what exists.
|
||
#[test]
|
||
fn the_liveness_probe_never_reports_a_frozen_rail_as_live() {
|
||
let run = production_probe();
|
||
let probed = run.opened.then(|| run.verdict());
|
||
let cached = crate::ioreport_status();
|
||
let oracle = oracle_cpu_rail(crate::CPU_RAIL_LIVENESS_BUDGET * 3);
|
||
eprintln!(
|
||
"ioreport probe: probed={probed:?} cached={cached:?} intervals={} over {:?} sample calls={} \
|
||
ticks={} advances per CPU rail {:?}; oracle={:?} ({} reads over {:?}, changes per CPU rail {:?}, \
|
||
per GPU rail {:?}, of {} compared pairs)",
|
||
run.watch.intervals,
|
||
run.watch.window,
|
||
run.calls.len(),
|
||
run.ticks,
|
||
run.watch.rails,
|
||
oracle.verdict,
|
||
oracle.reads,
|
||
oracle.took,
|
||
oracle.cpu_changes,
|
||
oracle.gpu_changes,
|
||
oracle.compared
|
||
);
|
||
oracle.assert_evidence();
|
||
if probed == Some(Some(CpuRail::Live)) || cached == IoReportStatus::Live {
|
||
assert_eq!(
|
||
oracle.verdict,
|
||
Oracle::CpuRailMoved(true),
|
||
"the probe said Live (probed={probed:?}, cached={cached:?}), the oracle did not see \
|
||
the CPU rail keep moving"
|
||
);
|
||
}
|
||
match oracle.verdict {
|
||
Oracle::NoIoReport => {
|
||
assert!(matches!(probed, None | Some(None)), "the oracle could not open IOReport, the probe did");
|
||
assert_eq!(cached, IoReportStatus::Unavailable);
|
||
}
|
||
Oracle::NoCpuRail => assert!(
|
||
matches!(probed, None | Some(None) | Some(Some(CpuRail::Absent)))
|
||
&& matches!(cached, IoReportStatus::CpuRailAbsent | IoReportStatus::Unavailable),
|
||
"the oracle found no CPU rail, the probe reported {probed:?} (cached={cached:?})"
|
||
),
|
||
Oracle::CpuRailMoved(false) => assert!(
|
||
probed == Some(Some(CpuRail::Frozen)) && cached == IoReportStatus::CpuRailFrozen,
|
||
"the CPU rail did not keep moving under the oracle's burn, so the probe must say Frozen: \
|
||
probed={probed:?}, cached={cached:?}"
|
||
),
|
||
Oracle::CpuRailMoved(true) => assert!(
|
||
matches!(probed, Some(Some(CpuRail::Live | CpuRail::Frozen))),
|
||
"the oracle saw every CPU rail keep moving but the probe reported {probed:?}"
|
||
),
|
||
}
|
||
}
|
||
|
||
/// The production probe burns CPU between samples, and exactly its tick
|
||
/// of it, pinned per tick on the probe's own records of this thread's
|
||
/// CPU clock (identified as such by the thread-clock test) so host load
|
||
/// cannot fail it: the CPU the thread consumed between the end of one
|
||
/// sample call and the start of the next must be at least half the
|
||
/// 25 ms tick (a probe that sleeps, or does nothing, between samples
|
||
/// burns ~0) and at most twice it (a slower cadence, e.g. a 100 ms tick,
|
||
/// burns more). And the tick CPU the probe recorded is checked against
|
||
/// the tests' own reading of the thread's CPU clock over the whole probe,
|
||
/// and every thread-clock reading the probe recorded must lie between the
|
||
/// tests' own readings of this thread's CPU clock just before and just
|
||
/// after it: the clock the production composition hands the probe is
|
||
/// this thread's CPU clock, not the wall clock or the process's.
|
||
#[test]
|
||
fn the_probe_burns_its_tick_of_thread_cpu_between_samples() {
|
||
let own0 = own_thread_cpu();
|
||
let started = Instant::now();
|
||
let run = production_probe();
|
||
let own1 = own_thread_cpu();
|
||
let own = own1.saturating_sub(own0);
|
||
let took = started.elapsed();
|
||
for (i, c) in run.calls.iter().enumerate() {
|
||
assert!(
|
||
own0 <= c.cpu_before && c.cpu_before <= c.cpu_after && c.cpu_after <= own1,
|
||
"sample call {i} recorded the probe's thread clock at {:?}..{:?}, outside this thread's own CPU \
|
||
clock readings {own0:?}..{own1:?} around the probe: the production probe is not timing this \
|
||
thread's CPU",
|
||
c.cpu_before,
|
||
c.cpu_after
|
||
);
|
||
}
|
||
if !run.opened {
|
||
let status = crate::ioreport_status();
|
||
eprintln!("ioreport probe tick: no Energy Model subscription opens here (status={status:?}): no probe to pin");
|
||
assert_eq!(status, IoReportStatus::Unavailable);
|
||
assert!(run.calls.is_empty() && run.ticks == 0, "{run:?}");
|
||
return;
|
||
}
|
||
let ticks: Vec<Duration> =
|
||
run.calls.windows(2).map(|w| w[1].cpu_before.saturating_sub(w[0].cpu_after)).collect();
|
||
let recorded: Duration = ticks.iter().sum();
|
||
eprintln!(
|
||
"ioreport probe tick: {} ticks of this thread's CPU between {} sample calls: min {:?}, median {:?}, \
|
||
max {:?} (specified 25 ms); the tests' own reading of the thread's CPU clock over the whole \
|
||
probe: {own:?} in {took:?} of wall time",
|
||
ticks.len(),
|
||
run.calls.len(),
|
||
ticks.iter().min(),
|
||
(!ticks.is_empty()).then(|| median(&ticks)),
|
||
ticks.iter().max()
|
||
);
|
||
assert!(ticks.len() >= 2, "the probe made only {} sample calls", run.calls.len());
|
||
assert_eq!(run.ticks as usize, ticks.len(), "every tick runs between two sample calls");
|
||
assert_eq!(run.tick_walls.len(), ticks.len(), "every tick's wall time is recorded");
|
||
for (i, t) in ticks.iter().enumerate() {
|
||
// A tick may burn less only where its wall cap cut it short: a
|
||
// host loaded past a hundred runnable threads gave this thread
|
||
// under 2% of a core for 500 ms (9.6 ms of CPU, load average
|
||
// 101). A tick that stopped short of both its CPU and its wall
|
||
// time waited or skipped, and fails.
|
||
// A tick cut short still spun whenever it ran, so it burned
|
||
// some CPU. A tick that slept through its 500 ms burned next to
|
||
// none, and fails either way.
|
||
if run.tick_walls[i] >= crate::clocks::PROBE_TICK_WALL_CAP {
|
||
eprintln!("ioreport probe tick {i}: {t:?} of CPU, cut short by its {:?} wall cap", run.tick_walls[i]);
|
||
assert!(
|
||
*t >= TICK / 25,
|
||
"tick {i} ran its whole {:?} wall cap and burned only {t:?} of this thread's CPU: it waited, \
|
||
not spun",
|
||
run.tick_walls[i]
|
||
);
|
||
continue;
|
||
}
|
||
assert!(
|
||
*t >= TICK / 2,
|
||
"tick {i} burned only {t:?} of this thread's CPU (every tick: {ticks:?}): the probe must burn \
|
||
CPU between samples, not wait or skip. A tick spins until the thread's CPU clock has \
|
||
advanced 25 ms, so host load cannot shorten it; only its 500 ms wall cap can, on a host \
|
||
that gives this thread under 5% of a core"
|
||
);
|
||
assert!(
|
||
*t <= TICK * 2,
|
||
"tick {i} burned {t:?} of this thread's CPU (every tick: {ticks:?}): the tick is 25 ms of CPU"
|
||
);
|
||
}
|
||
assert!(
|
||
own >= recorded,
|
||
"the probe recorded {recorded:?} of tick CPU, but this thread's CPU clock advanced only {own:?} \
|
||
over the whole probe"
|
||
);
|
||
}
|
||
|
||
/// The production probe samples the Energy Model group and nothing else,
|
||
/// pinned by structure, not time: every sample it took held exactly the
|
||
/// Energy Model group's channels — as many as the oracle's all-channel
|
||
/// read counts in that group, far fewer than every channel — and every
|
||
/// one of them is in the group. And, second, what a sample costs the
|
||
/// probing thread in CPU: the group takes ~6-9 ms of it on an M5 Max and
|
||
/// an M3 Ultra, every channel ~80-120 ms. No wall-clock bound: another
|
||
/// process's all-channel read can hold IOReport, and a group sample
|
||
/// then waits in wall time without costing this thread CPU.
|
||
#[test]
|
||
fn the_probe_samples_only_the_energy_model_group() {
|
||
let Some(oracle) = OracleSub::open() else {
|
||
let run = production_probe();
|
||
eprintln!("ioreport probe structure: no IOReport subscription opens here; probe opened={}", run.opened);
|
||
assert!(!run.opened && run.calls.is_empty(), "the oracle cannot open IOReport, the probe did: {run:?}");
|
||
assert_eq!(crate::ioreport_status(), IoReportStatus::Unavailable);
|
||
return;
|
||
};
|
||
let (all, group) = oracle.shape().expect("an oracle read");
|
||
let run = production_probe();
|
||
let taken: Vec<&SampleCall> = run.calls.iter().filter(|c| c.shape.is_some()).collect();
|
||
let mut shapes: Vec<(usize, bool)> = taken.iter().filter_map(|c| c.shape).collect();
|
||
shapes.dedup();
|
||
let cpus: Vec<Duration> = run.calls.iter().map(|c| c.cpu_after.saturating_sub(c.cpu_before)).collect();
|
||
let walls: Vec<Duration> = run.calls.iter().map(|c| c.wall).collect();
|
||
eprintln!(
|
||
"ioreport probe structure: {} of {} sample calls took a sample, shaped (channels, all Energy Model) \
|
||
{shapes:?}; the Energy Model group is {group} of the {all} channels in the oracle's all-channel \
|
||
read; thread CPU per sample call: median {:?} (limit 50 ms), max {:?}; wall per call (not \
|
||
asserted): median {:?}, max {:?}",
|
||
taken.len(),
|
||
run.calls.len(),
|
||
(!cpus.is_empty()).then(|| median(&cpus)),
|
||
cpus.iter().max(),
|
||
(!walls.is_empty()).then(|| median(&walls)),
|
||
walls.iter().max()
|
||
);
|
||
assert!(run.opened, "the oracle's all-channel subscription opened, the probe's group subscription did not");
|
||
assert!(taken.len() >= 2, "the probe took only {} samples", taken.len());
|
||
assert!(
|
||
group > 0 && group * 10 <= all,
|
||
"the oracle counts {group} Energy Model channels of {all}: the group is not far below every channel"
|
||
);
|
||
for (i, c) in taken.iter().enumerate() {
|
||
assert_eq!(
|
||
c.shape,
|
||
Some((group, true)),
|
||
"sample {i} held {:?} (channels, all in the Energy Model group), the group is {group} of {all} \
|
||
channels: the probe must sample its own subscription to the Energy Model group, not every \
|
||
channel",
|
||
c.shape
|
||
);
|
||
}
|
||
let cost = median(&cpus);
|
||
assert!(
|
||
cost < Duration::from_millis(50),
|
||
"the probe's sample calls took a median {cost:?} of this thread's CPU (every call: {cpus:?}): the \
|
||
Energy Model group costs ~6-9 ms, every channel ~80-120 ms"
|
||
);
|
||
}
|
||
|
||
/// The probe's sampler reads each CPU rail's true value. Around each of
|
||
/// three samples from the sampler the probe uses, on the probe's own
|
||
/// Energy Model subscription, the oracle reads the same rail names
|
||
/// (Energy Model group, absolute) just before and just after. A frozen
|
||
/// rail is constant, so where the oracle saw a rail hold across its two
|
||
/// reads the sampler's value must EQUAL it; where the rail moved, the
|
||
/// sampler's value must lie between the oracle's reads. The sampler must
|
||
/// also list exactly the oracle's CPU rails. A sampler that returned a
|
||
/// constant 0, a sum of rails, or another rail's value fails here.
|
||
#[test]
|
||
fn the_probe_sampler_reads_each_cpu_rail_as_the_oracle_does() {
|
||
let Some(state) = MacosIoReportCounter::open_energy_model() else {
|
||
let status = crate::ioreport_status();
|
||
eprintln!("ioreport sampler: no Energy Model subscription opens here (status={status:?})");
|
||
assert_eq!(status, IoReportStatus::Unavailable);
|
||
return;
|
||
};
|
||
let oracle = OracleSub::open().expect("the probe's Energy Model subscription opened, the oracle's did not");
|
||
let oracle_rails = || -> Vec<(String, i64)> {
|
||
oracle
|
||
.read()
|
||
.expect("an oracle read")
|
||
.into_iter()
|
||
.filter(|c| c.group == "Energy Model" && c.name.ends_with("CPU Energy"))
|
||
.map(|c| (c.name, c.raw))
|
||
.collect()
|
||
};
|
||
let names = |v: &[(String, i64)]| v.iter().map(|(n, _)| n.clone()).collect::<Vec<_>>();
|
||
let mut seen = Vec::new();
|
||
for _ in 0..3 {
|
||
let before = oracle_rails();
|
||
let sampled: Vec<(String, i64)> = state
|
||
.group_sample()
|
||
.expect("the probe's sampler returned no sample")
|
||
.values
|
||
.into_iter()
|
||
.filter(|(n, _)| n.ends_with("CPU Energy"))
|
||
.collect();
|
||
let after = oracle_rails();
|
||
assert_eq!(names(&before), names(&after), "the oracle's CPU rails changed between two reads");
|
||
assert_eq!(
|
||
names(&sampled),
|
||
names(&before),
|
||
"the probe's sampler does not list the CPU rails the oracle reads"
|
||
);
|
||
for ((name, got), ((_, lo), (_, hi))) in sampled.iter().zip(before.iter().zip(&after)) {
|
||
if lo == hi {
|
||
assert_eq!(
|
||
got, lo,
|
||
"{name} held at {lo} across the oracle's reads either side of the sample, and the probe's \
|
||
sampler read {got}"
|
||
);
|
||
} else {
|
||
assert!(
|
||
(*lo.min(hi)..=*lo.max(hi)).contains(got),
|
||
"{name} moved {lo} -> {hi} across the oracle's reads either side of the sample, and the \
|
||
probe's sampler read {got}, outside them"
|
||
);
|
||
}
|
||
seen.push(format!("{name}: oracle {lo}..{hi}, sampler {got}"));
|
||
}
|
||
}
|
||
eprintln!("ioreport sampler: {} CPU-rail values checked against the oracle: {seen:?}", seen.len());
|
||
}
|
||
|
||
/// What the production composition hands the rule is what the rails
|
||
/// hold. The oracle reads the Energy Model group's CPU and GPU energy
|
||
/// rails, the production probe runs (uncounted, uncached), and the
|
||
/// oracle reads again. In every sample the probe took: its rails are
|
||
/// exactly the oracle's; a CPU rail the oracle saw hold must EQUAL the
|
||
/// oracle's value, and one that moved must lie within the oracle's two
|
||
/// reads; so must GPU Energy, which moves on these hosts, so a sampler
|
||
/// replaying a sample from before the bracket fails. And where the
|
||
/// oracle saw GPU Energy change across the bracket, the probe's samples
|
||
/// must show it changing too, so a sampler replaying its own first
|
||
/// sample fails; where the oracle saw it hold, that is not asserted, and
|
||
/// the line says so.
|
||
#[test]
|
||
fn the_production_probe_hands_the_rule_what_the_oracle_reads() {
|
||
let Some(oracle) = OracleSub::open() else {
|
||
let run = production_probe();
|
||
eprintln!("ioreport freshness: no IOReport subscription opens here; probe opened={}", run.opened);
|
||
assert!(!run.opened && run.calls.is_empty(), "the oracle cannot open IOReport, the probe did: {run:?}");
|
||
return;
|
||
};
|
||
let before = oracle.energy_rails();
|
||
let run = production_probe();
|
||
let after = oracle.energy_rails();
|
||
let keys = |v: &[(RailKey, i64)]| {
|
||
let mut k: Vec<RailKey> = v.iter().map(|(k, _)| k.clone()).collect();
|
||
k.sort();
|
||
k
|
||
};
|
||
assert_eq!(keys(&before), keys(&after), "the oracle's rails changed between its two reads");
|
||
let bracket = |key: &RailKey| -> (i64, i64) {
|
||
let find = |v: &[(RailKey, i64)]| v.iter().find(|(k, _)| k == key).map(|(_, raw)| *raw);
|
||
(find(&before).expect("a rail the oracle read"), find(&after).expect("a rail the oracle read"))
|
||
};
|
||
assert!(run.opened, "the oracle's all-channel subscription opened, the probe's group subscription did not");
|
||
let handed: Vec<&Vec<(RailKey, i64)>> = run.calls.iter().filter_map(|c| c.handed.as_ref()).collect();
|
||
assert!(handed.len() >= 2, "the probe took only {} samples", handed.len());
|
||
for (i, sample) in handed.iter().enumerate() {
|
||
assert_eq!(
|
||
keys(sample),
|
||
keys(&before),
|
||
"sample {i}: the production probe handed the rule these CPU and GPU rails, the oracle reads others"
|
||
);
|
||
for (key, got) in sample.iter() {
|
||
let (lo, hi) = bracket(key);
|
||
if lo == hi {
|
||
assert_eq!(
|
||
*got, lo,
|
||
"sample {i}: {key:?} held at {lo} across the oracle's reads either side of the production \
|
||
probe, and the probe handed the rule {got}"
|
||
);
|
||
} else {
|
||
assert!(
|
||
(lo.min(hi)..=lo.max(hi)).contains(got),
|
||
"sample {i}: {key:?} moved {lo} -> {hi} across the oracle's reads either side of the \
|
||
production probe, and the probe handed the rule {got}, outside them"
|
||
);
|
||
}
|
||
}
|
||
}
|
||
let mut lines = Vec::new();
|
||
for (key, lo) in before.iter().filter(|((name, _), _)| name.ends_with("GPU Energy")) {
|
||
let hi = bracket(key).1;
|
||
let mut values: Vec<i64> =
|
||
handed.iter().filter_map(|s| s.iter().find(|(k, _)| k == key).map(|(_, v)| *v)).collect();
|
||
values.dedup();
|
||
if *lo != hi {
|
||
// The oracle's two reads *bracket* the probe's samples, so a rise it
|
||
// saw can fall entirely outside the probe's own window. GPU energy
|
||
// rises only while the GPU works -- it is activity-driven, not a clock
|
||
// -- so on a shared host another process's short burst lands between
|
||
// the oracle's first read and the probe's first sample as easily as
|
||
// inside it. One value here is therefore no evidence of a sampler that
|
||
// does not sample, and asserting it was a rise the probe must also see
|
||
// failed once in ~300 CI runs on an otherwise green tree.
|
||
//
|
||
// Nothing is lost by not asserting it here: that the sampler re-reads
|
||
// is already pinned deterministically, per sample, by the scripted
|
||
// probes -- `probe_with_a_single_batch_jump_is_frozen` drives
|
||
// `GPU Energy` to a different value at every sample and asserts each
|
||
// one is what the rule was handed, as does
|
||
// `probe_with_one_die_live_and_one_frozen_is_frozen`. A sampler that
|
||
// cached one reading fails those on sample 1, on every host. So do
|
||
// not restore a hardware assertion here believing this is uncovered:
|
||
// no rail on the CI host reliably moves inside the probe's window
|
||
// (its CPU rails are frozen), so there is no sound hardware witness
|
||
// to assert on.
|
||
lines.push(if values.len() >= 2 {
|
||
format!("{}: moved {lo} -> {hi}, the probe read {} distinct values within it", key.0, values.len())
|
||
} else {
|
||
format!(
|
||
"{}: moved {lo} -> {hi} across the oracle's reads, but the probe read only {values:?}, \
|
||
so the rise fell outside its window and freshness is NOT asserted on it",
|
||
key.0
|
||
)
|
||
});
|
||
} else {
|
||
lines.push(format!("{}: held at {lo} across the bracket, so freshness is NOT asserted on it", key.0));
|
||
}
|
||
}
|
||
let cpu: Vec<String> = before
|
||
.iter()
|
||
.filter(|((name, _), _)| name.ends_with("CPU Energy"))
|
||
.map(|(key, lo)| format!("{}: {lo}..{}", key.0, bracket(key).1))
|
||
.collect();
|
||
eprintln!(
|
||
"ioreport freshness: {} samples from the production probe, each handing the rule the oracle's rails; \
|
||
CPU rails (oracle before..after) {cpu:?}; {lines:?}",
|
||
handed.len()
|
||
);
|
||
assert!(!lines.is_empty(), "the oracle read no GPU Energy rail in the Energy Model group");
|
||
}
|
||
|
||
/// A rise of `raw` in `unit`, in whole µJ, rounded down, by the oracle's
|
||
/// own table: `None` for a unit it has no exact conversion for. A fall
|
||
/// is no rise.
|
||
fn exact_uj(raw: i64, unit: &str) -> Option<u64> {
|
||
let raw = raw.max(0) as u64;
|
||
match unit {
|
||
"pJ" => Some(raw / 1_000_000),
|
||
"nJ" => Some(raw / 1_000),
|
||
"uJ" | "µJ" => Some(raw),
|
||
"mJ" => Some(raw.saturating_mul(1_000)),
|
||
"J" => Some(raw.saturating_mul(1_000_000)),
|
||
_ => None,
|
||
}
|
||
}
|
||
|
||
/// Every Energy Model channel the counter sums (by [`counts_once`]), as
|
||
/// the oracle reads it: ((name, ordinal), (unit, absolute raw)).
|
||
fn summed_rails(oracle: &OracleSub) -> Vec<(RailKey, (String, i64))> {
|
||
let (_, chans) = oracle.read_where(|group, _| group == "Energy Model").expect("an oracle read");
|
||
oracle_keyed(chans.into_iter().filter(|c| counts_once(&c.name)).map(|c| (c.name, (c.unit, c.raw))))
|
||
}
|
||
|
||
/// What the summed rails rose from one oracle read to another, rail by
|
||
/// rail, rounded down to whole µJ per rail as the counter rounds.
|
||
fn rose_uj(from: &[(RailKey, (String, i64))], to: &[(RailKey, (String, i64))]) -> u64 {
|
||
assert_eq!(
|
||
from.iter().map(|(k, _)| k).collect::<Vec<_>>(),
|
||
to.iter().map(|(k, _)| k).collect::<Vec<_>>(),
|
||
"the summed rails changed between two oracle reads"
|
||
);
|
||
from.iter()
|
||
.zip(to)
|
||
.map(|((key, (unit, a)), (_, (_, b)))| {
|
||
exact_uj(b - a, unit).unwrap_or_else(|| panic!("{key:?} is in {unit:?}, which the oracle cannot convert"))
|
||
})
|
||
.sum()
|
||
}
|
||
|
||
/// The counter's own read path, which a declined host never runs: built
|
||
/// past the liveness gate on purpose (given `Live`), it must report
|
||
/// exactly what its rails did. The oracle reads every rail the counter
|
||
/// sums (its own subscription to the Energy Model group, absolute
|
||
/// values, ~7 ms a read) just before and just after the counter's
|
||
/// baseline read, and again just before and just after its second read,
|
||
/// holding the window between them open for at least 3 s and until the
|
||
/// summed rails rose (GPU Energy moves on these hosts), for at most 5 s.
|
||
/// The rails are cumulative, so the counter's delta must lie between
|
||
/// their rise from the second oracle read to the third (inside the
|
||
/// counter's window) and from the first to the fourth (around it) —
|
||
/// rail by rail, rounded down as the counter rounds. Nothing moved:
|
||
/// 0 == 0, still asserted. A counter that left a moving rail out lands
|
||
/// below the bracket. The bracket is wider than the window by the
|
||
/// counter's own two reads (every channel: 0.1 s alone, over 1 s while
|
||
/// other processes hold IOReport), so one that summed a moving rail
|
||
/// twice lands above it only while those two reads together take less
|
||
/// than the window.
|
||
#[test]
|
||
fn the_counter_advances_when_a_rail_it_sums_advanced() {
|
||
let counter = match MacosIoReportCounter::try_detect_given(IoReportStatus::Live) {
|
||
Ok(counter) => counter,
|
||
Err(e) => {
|
||
let selected = crate::detect().name() == "ioreport";
|
||
eprintln!("ioreport counter: no IOReport subscription opens here ({e:?}); selected={selected}");
|
||
assert!(matches!(e, Error::NoCounter), "declined with {e:?}");
|
||
assert!(OracleSub::open().is_none(), "the counter's all-channel subscription did not open, the oracle's did");
|
||
assert!(!selected, "detect() selected an IOReport counter that does not open");
|
||
return;
|
||
}
|
||
};
|
||
let oracle = OracleSub::open_group().expect("the counter's subscription opened, the oracle's group one did not");
|
||
let o1 = summed_rails(&oracle);
|
||
let base = counter.read_microjoules().expect("the counter's baseline read");
|
||
let o2 = summed_rails(&oracle);
|
||
let started = Instant::now();
|
||
let mut o3 = summed_rails(&oracle);
|
||
while started.elapsed() < Duration::from_secs(3)
|
||
|| (rose_uj(&o2, &o3) == 0 && started.elapsed() < Duration::from_secs(5))
|
||
{
|
||
std::thread::sleep(Duration::from_millis(20));
|
||
o3 = summed_rails(&oracle);
|
||
}
|
||
let after = counter.read_microjoules().expect("the counter's second read");
|
||
let o4 = summed_rails(&oracle);
|
||
let (inner, outer) = (rose_uj(&o2, &o3), rose_uj(&o1, &o4));
|
||
let delta = after.saturating_sub(base);
|
||
let moved: Vec<String> = o1
|
||
.iter()
|
||
.zip(&o4)
|
||
.filter(|((_, (_, a)), (_, (_, b)))| a != b)
|
||
.map(|((key, (unit, a)), (_, (_, b)))| format!("{} +{} {unit}", key.0, b - a))
|
||
.collect();
|
||
eprintln!(
|
||
"ioreport counter: {} rails summed; across the bracket they rose {inner} µJ inside the counter's \
|
||
window and {outer} µJ around it (moved: {moved:?}); the counter read {base} -> {after} µJ = {delta} µJ",
|
||
o1.len()
|
||
);
|
||
assert!(after >= base, "the counter went backwards: {base} -> {after} µJ");
|
||
assert!(
|
||
(inner..=outer).contains(&delta),
|
||
"the counter read {delta} µJ over its window, but the rails it sums rose {inner} µJ inside that window \
|
||
and {outer} µJ around it (moved: {moved:?}): it did not sum what its rails did"
|
||
);
|
||
}
|
||
|
||
/// When the counter IS available, a CPU burn meters as non-zero energy
|
||
/// through a rail that moves. When it is not, the status is never `Live`,
|
||
/// and what [`crate::detect`] falls back to meters real work as no figure
|
||
/// at all — its reads fail, or it is the stub, whose zero every consumer
|
||
/// treats as no reading — so each caller keeps its own estimate: never a
|
||
/// hardware-tagged number, and never a zero passed off as a reading.
|
||
#[test]
|
||
fn a_selected_ioreport_counter_meters_cpu_work() {
|
||
match MacosIoReportCounter::try_detect() {
|
||
Ok(c) => {
|
||
assert_eq!(
|
||
crate::ioreport_status(),
|
||
IoReportStatus::Live,
|
||
"try_detect() constructed the counter although the status is not Live"
|
||
);
|
||
let rail = production_probe().verdict();
|
||
assert_eq!(rail, Some(CpuRail::Live), "counter available but its CPU rail is {rail:?}");
|
||
let _ = c.read_microjoules(); // first read only sets the baseline sample
|
||
let before = c.read_microjoules().unwrap();
|
||
spin(Duration::from_millis(200));
|
||
let after = c.read_microjoules().unwrap();
|
||
assert!(
|
||
after > before,
|
||
"a live CPU rail metered 200 ms of CPU work as zero: before={before} after={after}"
|
||
);
|
||
}
|
||
Err(e) => {
|
||
let status = crate::ioreport_status();
|
||
assert!(matches!(e, Error::NoCounter), "declined with {e:?}, expected NoCounter");
|
||
assert_ne!(status, IoReportStatus::Live, "status Live but try_detect() declined");
|
||
let fallback = crate::detect();
|
||
let before = fallback.read_microjoules();
|
||
spin(Duration::from_millis(100));
|
||
let after = fallback.read_microjoules();
|
||
eprintln!(
|
||
"ioreport declined ({status:?}): fallback={} read {before:?} then {after:?} across \
|
||
100 ms of CPU work",
|
||
fallback.name()
|
||
);
|
||
assert_ne!(fallback.name(), "ioreport", "detect() fell back to the declined counter");
|
||
if fallback.name() == "stub" {
|
||
assert!(matches!((&before, &after), (Ok(0), Ok(0))), "the stub read {before:?}, {after:?}");
|
||
} else {
|
||
assert!(
|
||
before.is_err() && after.is_err(),
|
||
"the fallback counter {} returned a figure ({before:?}, {after:?}) though no \
|
||
hardware counter measured the work",
|
||
fallback.name()
|
||
);
|
||
}
|
||
}
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
#[cfg(target_os = "macos")]
|
||
pub use macos_ioreport::MacosIoReportCounter;
|
||
|
||
/// Why the IOReport counter was or was not made available on this host.
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||
pub enum IoReportStatus {
|
||
/// No subscription to IOReport's Energy Model group could be opened (not
|
||
/// macOS, not Apple Silicon, Rosetta, or no such group), or one opened but
|
||
/// never yielded two usable samples, so nothing could be learned about the
|
||
/// CPU rail.
|
||
Unavailable,
|
||
/// IOReport's Energy Model group was sampled, and no `CPU Energy` rail was
|
||
/// in it — an unfamiliar channel naming. The counter is not used.
|
||
CpuRailAbsent,
|
||
/// A CPU energy rail exists but did not keep advancing while CPU was
|
||
/// burned: it held, or stepped only in a burst (a refresh, which macOS 27
|
||
/// can publish in two or more steps a few tens of milliseconds apart). On
|
||
/// a multi-die part, one die's frozen rail is enough, even beside a live
|
||
/// one: the counter sums every die, and would claim the frozen die's
|
||
/// energy as measured.
|
||
/// The platform is not exposing live CPU energy to this process, so using
|
||
/// the counter would report GPU-only energy, zero, or a stale refresh as
|
||
/// the energy of CPU work. Observed on macOS 27.0.1 on both an M5 Max and
|
||
/// an M3 Ultra.
|
||
CpuRailFrozen,
|
||
/// Every CPU energy rail (each die's, on a multi-die part) kept advancing
|
||
/// under load — each in at least five separate sample intervals, at least
|
||
/// 80% of those observed, over at least 300 ms of wall time; the counter is
|
||
/// available.
|
||
Live,
|
||
}
|
||
|
||
/// How long, in WALL time, the liveness probe keeps sampling before it
|
||
/// concludes the CPU rail is not live. It spends that second burning CPU on
|
||
/// the calling thread in ticks of 25 ms of that thread's own CPU time,
|
||
/// sampling the rail after each (a tick that cannot get its CPU ends after
|
||
/// 500 ms of wall time). A loaded host gives the probing thread less of a core
|
||
/// in the same second, so the probe sees fewer intervals (on an M5 Max under
|
||
/// 1 × ncpu busy spinners, often half as many or fewer), and a live rail has
|
||
/// fewer chances to show a sustained advance: five advancing intervals, at
|
||
/// least 80% of those observed, over at least 300 ms. That fails safe:
|
||
/// declined, so callers estimate; never over-claimed. If sampling is so slow
|
||
/// that fewer than five intervals fit in the budget, the probe keeps going
|
||
/// until five have, and never past five budgets. A live rail ends the probe
|
||
/// early, after 300 ms at the least. Paid once per process; see
|
||
/// [`ioreport_status`].
|
||
#[cfg(target_os = "macos")]
|
||
pub const CPU_RAIL_LIVENESS_BUDGET: std::time::Duration = std::time::Duration::from_millis(1000);
|
||
|
||
/// This host's IOReport status, probed once per process and cached.
|
||
///
|
||
/// `MacosIoReportCounter::try_detect` (and so [`detect`]) makes the IOReport
|
||
/// counter available only when this is [`IoReportStatus::Live`]. The first
|
||
/// call on macOS burns CPU on the calling thread for about
|
||
/// `CPU_RAIL_LIVENESS_BUDGET` of wall time (less when the rail proves live,
|
||
/// though never under 300 ms; at most five budgets). Exposed so a caller or CLI can say *why* an Apple
|
||
/// Silicon host is not metered by IOReport.
|
||
pub fn ioreport_status() -> IoReportStatus {
|
||
#[cfg(target_os = "macos")]
|
||
{
|
||
// Test builds only: a status a test installed on this thread stands in
|
||
// for the cached one.
|
||
#[cfg(test)]
|
||
if let Some(status) = macos_ioreport::script::status() {
|
||
return status;
|
||
}
|
||
static STATUS: std::sync::OnceLock<IoReportStatus> = std::sync::OnceLock::new();
|
||
*STATUS.get_or_init(macos_ioreport::host_status)
|
||
}
|
||
#[cfg(not(target_os = "macos"))]
|
||
{
|
||
IoReportStatus::Unavailable
|
||
}
|
||
}
|
||
|
||
/// Test support, not API: how many times the liveness probe behind
|
||
/// [`ioreport_status`] has run in this process (always 0 off macOS). The
|
||
/// cache must keep it at 1 however many threads ask, construct or select.
|
||
#[doc(hidden)]
|
||
pub fn liveness_probe_runs() -> usize {
|
||
#[cfg(target_os = "macos")]
|
||
{
|
||
macos_ioreport::HOST_STATUS_RUNS.load(std::sync::atomic::Ordering::SeqCst)
|
||
}
|
||
#[cfg(not(target_os = "macos"))]
|
||
{
|
||
0
|
||
}
|
||
}
|
||
|
||
/// Test support, not API: how many times this process has opened the
|
||
/// liveness probe's Energy Model sampler, whoever called it (always 0 off
|
||
/// macOS). A probe run outside the cache opens one too.
|
||
#[doc(hidden)]
|
||
pub fn liveness_sampler_opens() -> usize {
|
||
#[cfg(target_os = "macos")]
|
||
{
|
||
macos_ioreport::SAMPLER_OPENS.load(std::sync::atomic::Ordering::SeqCst)
|
||
}
|
||
#[cfg(not(target_os = "macos"))]
|
||
{
|
||
0
|
||
}
|
||
}
|
||
|
||
|
||
/// The liveness of a single vendor energy counter rung — RAPL's package
|
||
/// counter, NVML's total-energy counter — as its probe found it on this host,
|
||
/// by the rule IOReport's CPU rail is held to. Probed once per process and
|
||
/// cached; see [`rapl_status`] and [`nvml_status`]. Only
|
||
/// [`CounterStatus::Live`] admits the rung: a live vendor counter's reading is
|
||
/// a measurement, and nothing else from it is.
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||
pub enum CounterStatus {
|
||
/// The counter could not be opened or read — not this platform, the
|
||
/// feature not built, no such device, no RAPL zone named `package-<n>`
|
||
/// (a host that lists only `psys`), a locked-down sysfs (`energy_uj` is
|
||
/// root-only on most current kernels) — or it gave fewer than two usable
|
||
/// samples in five budgets. Never selected.
|
||
Unavailable,
|
||
/// The counter read, but did not keep advancing while the probe ran: it
|
||
/// held, or was republished in lumps. Never selected: a lump landing in
|
||
/// one caller's span would be sealed as that span's reading.
|
||
Frozen,
|
||
/// The counter kept advancing, but a GPU counter (NVML's) ran faster than
|
||
/// its board can draw power over the probe: past 1.5 times the board's
|
||
/// maximum power limit, or past twice the board's own power draw plus
|
||
/// 5 W. It is not counting the board's energy. Seen on an RTX 4050 Laptop
|
||
/// GPU (driver 595.84), whose counter runs at 4 to 30 times the board's
|
||
/// draw while it is polled. Never constructed.
|
||
Implausible,
|
||
/// The counter kept advancing and could not be judged, because the board's
|
||
/// own power telemetry was unusable over the probe: a draw reading past
|
||
/// the board's own maximum power limit (an RTX 4050 Laptop GPU reports
|
||
/// about 590 W for a second after it wakes).
|
||
///
|
||
/// Split out from [`CounterStatus::Implausible`], which used to cover this
|
||
/// — its doc then asserted "it is not counting the board's energy", a
|
||
/// conclusion this case does not support. The yardstick is broken, not
|
||
/// necessarily the counter. Never selected either way: unjudgeable is not
|
||
/// admissible, and erring toward declining is the safe direction. The
|
||
/// distinction is the diagnosis an operator gets, which is the difference
|
||
/// between replacing a counter and waiting for the board to settle.
|
||
Unjudgeable,
|
||
/// The counter kept advancing while the probe ran: in at least five
|
||
/// separate sample intervals, at least 80% of those observed, over at
|
||
/// least 300 ms. The rung is selected.
|
||
Live,
|
||
}
|
||
|
||
/// How long, in wall time, a vendor counter's liveness probe watches a counter
|
||
/// that is not live before declining it: one second, as
|
||
/// `CPU_RAIL_LIVENESS_BUDGET`. A live counter is called live about 300 ms in.
|
||
/// Paid once per process, and only where the counter opens.
|
||
#[cfg_attr(not(any(target_os = "linux", target_os = "macos", feature = "cuda")), allow(dead_code))] // used by RAPL, NVML and the process counter
|
||
const VENDOR_COUNTER_LIVENESS_BUDGET: std::time::Duration = std::time::Duration::from_millis(1000);
|
||
|
||
/// This host's RAPL status, probed once per process and cached.
|
||
///
|
||
/// [`detect`] selects the RAPL rung only when this is [`CounterStatus::Live`]:
|
||
/// every package counter — each top-level powercap zone whose `name` reads
|
||
/// `package-<n>`, so each socket of a multi-socket server, and never `psys`,
|
||
/// `dram` or a sub-zone, wherever it sits — read, and each kept advancing
|
||
/// while the probing thread burned CPU, in at least five sample intervals, at
|
||
/// least 80% of them, over at least 300 ms. The counter's figure sums them
|
||
/// all, so one that holds, or is republished in lumps (a virtualised powercap
|
||
/// tree can do either), declines the rung. Where there is no package zone, or
|
||
/// a package counter is unreadable, it is [`CounterStatus::Unavailable`]
|
||
/// without burning anything. Always `Unavailable` off Linux. The first call on
|
||
/// a host whose counters read burns about 300 ms of the calling thread's CPU
|
||
/// (a second if they are not live); the package set it watched is cached
|
||
/// beside the status, and is the set the counter sums.
|
||
pub fn rapl_status() -> CounterStatus {
|
||
#[cfg(target_os = "linux")]
|
||
{
|
||
linux_rapl::host().status
|
||
}
|
||
#[cfg(not(target_os = "linux"))]
|
||
{
|
||
CounterStatus::Unavailable
|
||
}
|
||
}
|
||
|
||
/// This host's NVML status, probed once per process and cached. Always
|
||
/// [`CounterStatus::Unavailable`] without the `cuda` feature.
|
||
///
|
||
/// [`NvmlCounter`] is constructed only when this is [`CounterStatus::Live`]:
|
||
/// GPU 0's total-energy counter kept advancing, in at least five sample
|
||
/// intervals, at least 80% of them, over at least 300 ms, and over that window
|
||
/// counted no faster than the board can draw (with the board's power draw read
|
||
/// at every sample and its maximum power limit; otherwise
|
||
/// [`CounterStatus::Implausible`]). The probe cannot load the GPU, so it waits
|
||
/// between samples rather than burning CPU, and a powered GPU's own draw
|
||
/// advances a live counter; this shows the counter is published continuously,
|
||
/// not that it tracks a load. One that holds or steps in lumps is declined.
|
||
/// [`detect`] never selects the GPU counter, live or not. The first call where
|
||
/// NVML opens waits about half a second (a second if the counter is not live).
|
||
pub fn nvml_status() -> CounterStatus {
|
||
#[cfg(feature = "cuda")]
|
||
{
|
||
static STATUS: std::sync::OnceLock<CounterStatus> = std::sync::OnceLock::new();
|
||
*STATUS.get_or_init(nvml::host_status)
|
||
}
|
||
#[cfg(not(feature = "cuda"))]
|
||
{
|
||
CounterStatus::Unavailable
|
||
}
|
||
}
|
||
|
||
// Compiled for tests on every platform, so its zone listing, sampler, counter
|
||
// and gate run under test on a Mac too, over fake powercap directories;
|
||
// selected only on Linux.
|
||
#[cfg(any(target_os = "linux", test))]
|
||
#[cfg_attr(not(target_os = "linux"), allow(dead_code))]
|
||
mod linux_rapl {
|
||
use super::*;
|
||
use crate::rail_liveness;
|
||
use std::path::{Path, PathBuf};
|
||
use std::sync::Mutex;
|
||
use std::time::Duration;
|
||
|
||
/// Where the kernel lists its powercap zones, RAPL's among them.
|
||
pub(crate) const POWERCAP_ROOT: &str = "/sys/class/powercap";
|
||
|
||
/// One RAPL package zone: a top-level `intel-rapl:<index>` zone whose
|
||
/// `name` reads `package-<n>`.
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||
pub(crate) struct Package {
|
||
/// `<index>` in `intel-rapl:<index>`.
|
||
pub(crate) index: u32,
|
||
/// The zone's `max_energy_range_uj`, the value its `energy_uj` wraps
|
||
/// at, when that reads as a number above zero.
|
||
pub(crate) max_energy_range_uj: Option<u64>,
|
||
}
|
||
|
||
/// Whether a top-level zone's `name` file names a package: `package-<n>`,
|
||
/// `<n>` one or more digits (the file ends in a newline). Not `psys` (the
|
||
/// platform zone, whose energy includes the packages'), `dram`, `core`
|
||
/// or `uncore`.
|
||
pub(crate) fn is_package_name(name: &str) -> bool {
|
||
name.trim().strip_prefix("package-").is_some_and(|n| !n.is_empty() && n.bytes().all(|b| b.is_ascii_digit()))
|
||
}
|
||
|
||
/// The index of a top-level RAPL zone directory, `intel-rapl:<index>`.
|
||
/// `None` for anything else: a sub-zone (`intel-rapl:0:0`, a package's
|
||
/// core, uncore or dram), the control type itself (`intel-rapl`), and the
|
||
/// MMIO interface (`intel-rapl-mmio:0`, also named `package-0`, which
|
||
/// republishes a package's energy and would count it twice).
|
||
pub(crate) fn top_level_zone(dir: &str) -> Option<u32> {
|
||
let n = dir.strip_prefix("intel-rapl:")?;
|
||
if n.is_empty() || !n.bytes().all(|b| b.is_ascii_digit()) {
|
||
return None;
|
||
}
|
||
n.parse().ok()
|
||
}
|
||
|
||
/// Zone `index`'s directory under the powercap directory `root`.
|
||
fn zone_dir(root: &Path, index: u32) -> PathBuf {
|
||
root.join(format!("intel-rapl:{index}"))
|
||
}
|
||
|
||
/// The file at `path`, read and parsed as an unsigned integer; `None` when
|
||
/// it cannot be.
|
||
pub(crate) fn read_u64(path: &Path) -> Option<u64> {
|
||
std::fs::read_to_string(path).ok()?.trim().parse::<u64>().ok()
|
||
}
|
||
|
||
/// The package zones under the powercap directory `root`, in index order:
|
||
/// every top-level zone whose `name` reads `package-<n>`, with the range
|
||
/// its counter wraps at. Everything else is left out wherever it sits,
|
||
/// zone 0 included: `psys`, a `dram` zone, sub-zones, the MMIO interface.
|
||
/// Empty when no zone is a package, or `root` cannot be listed.
|
||
pub(crate) fn packages_under(root: &Path) -> Vec<Package> {
|
||
let Ok(entries) = std::fs::read_dir(root) else {
|
||
return Vec::new();
|
||
};
|
||
let mut packages: Vec<Package> = entries
|
||
.flatten()
|
||
.filter_map(|entry| {
|
||
let index = top_level_zone(entry.file_name().to_str()?)?;
|
||
let dir = entry.path();
|
||
let name = std::fs::read_to_string(dir.join("name")).ok()?;
|
||
is_package_name(&name).then(|| Package {
|
||
index,
|
||
max_energy_range_uj: read_u64(&dir.join("max_energy_range_uj")).filter(|&m| m > 0),
|
||
})
|
||
})
|
||
.collect();
|
||
packages.sort_by_key(|p| p.index);
|
||
packages
|
||
}
|
||
|
||
/// Every package's `energy_uj` under `root`, in `packages` order. `None`
|
||
/// when there is no package, or when any one cannot be read: a figure that
|
||
/// left a package out would miss its energy while claiming the whole CPU.
|
||
pub(crate) fn read_packages(root: &Path, packages: &[Package]) -> Option<Vec<u64>> {
|
||
if packages.is_empty() {
|
||
return None;
|
||
}
|
||
packages.iter().map(|p| read_u64(&zone_dir(root, p.index).join("energy_uj"))).collect()
|
||
}
|
||
|
||
/// One zone's energy between two reads, modulo its own wrap. A counter
|
||
/// that went down wrapped at `max_range`; without a known (non-zero)
|
||
/// range, or with an earlier read above it, the energy is unknown and
|
||
/// `None`. Undercounts a wrap by at most one counter unit, and cannot see
|
||
/// more than one wrap between two reads: about a quarter of an hour at
|
||
/// 300 W for a package's usual 262,143,328,850 µJ range.
|
||
pub(crate) fn zone_delta_uj(e0: u64, e1: u64, max_range: Option<u64>) -> Option<u64> {
|
||
if e1 >= e0 {
|
||
return Some(e1 - e0);
|
||
}
|
||
max_range.filter(|&m| m > 0)?.checked_sub(e0)?.checked_add(e1)
|
||
}
|
||
|
||
/// A RAPL counter's running total across its reads.
|
||
#[derive(Debug)]
|
||
pub(crate) enum Running {
|
||
/// Not read yet.
|
||
Unread,
|
||
/// Each package's raw `energy_uj` at the last read, and the total so
|
||
/// far: the first read's counters summed, plus every package's energy
|
||
/// since, each taken modulo its own range.
|
||
At { last: Vec<u64>, total: u64 },
|
||
/// A package's energy between two reads could not be known: its
|
||
/// counter went down and its range is unknown. The total no longer
|
||
/// holds that energy, so every read from then on fails rather than
|
||
/// hand out a figure short by it.
|
||
Lost(String),
|
||
}
|
||
|
||
impl Running {
|
||
/// Fold in `now`, every package's raw counter read in `packages` order,
|
||
/// and return the running total. Each package's wrap is taken at ITS
|
||
/// OWN range; summing the raw counters instead would turn one
|
||
/// package's wrap into a fall of the sum.
|
||
pub(crate) fn advance(&mut self, packages: &[Package], now: &[u64]) -> Result<u64> {
|
||
match self {
|
||
Running::Lost(why) => Err(Error::Backend(why.clone())),
|
||
Running::Unread => {
|
||
let total = now
|
||
.iter()
|
||
.try_fold(0u64, |sum, &uj| sum.checked_add(uj))
|
||
.ok_or_else(|| Error::Backend("the RAPL package counters' sum overflows u64".into()))?;
|
||
*self = Running::At { last: now.to_vec(), total };
|
||
Ok(total)
|
||
}
|
||
Running::At { last, total } => {
|
||
if last.len() != now.len() || packages.len() != now.len() {
|
||
return Err(Error::Backend("a RAPL read did not cover every package".into()));
|
||
}
|
||
match energy_between(packages, last, now).and_then(|energy| {
|
||
total.checked_add(energy).ok_or_else(|| "the RAPL running total overflows u64".to_string())
|
||
}) {
|
||
Ok(sum) => {
|
||
*total = sum;
|
||
last.copy_from_slice(now);
|
||
Ok(sum)
|
||
}
|
||
Err(why) => {
|
||
*self = Running::Lost(why.clone());
|
||
Err(Error::Backend(why))
|
||
}
|
||
}
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
/// Every package's energy between the raw counters `last` and `now`,
|
||
/// summed, each taken modulo its own range ([`zone_delta_uj`]); why not,
|
||
/// when one package's energy is unknown.
|
||
fn energy_between(packages: &[Package], last: &[u64], now: &[u64]) -> std::result::Result<u64, String> {
|
||
let mut energy = 0u64;
|
||
for ((p, &e0), &e1) in packages.iter().zip(last).zip(now) {
|
||
let delta = zone_delta_uj(e0, e1, p.max_energy_range_uj).ok_or_else(|| {
|
||
format!(
|
||
"RAPL zone intel-rapl:{} went down from {e0} to {e1} uJ with no readable \
|
||
max_energy_range_uj; its energy between the two reads is unknown",
|
||
p.index
|
||
)
|
||
})?;
|
||
energy = energy.checked_add(delta).ok_or_else(|| "RAPL package energy overflows u64".to_string())?;
|
||
}
|
||
Ok(energy)
|
||
}
|
||
|
||
/// Reads RAPL's package energy through powercap: the sum over every
|
||
/// package zone (`intel-rapl:<n>` named `package-<n>`), each zone's wrap
|
||
/// taken at its own `max_energy_range_uj`, as a running total.
|
||
///
|
||
/// Constructed only on a host whose package counters the liveness probe
|
||
/// found live, every one of them ([`crate::rapl_status`]); a live RAPL
|
||
/// counter is a vendor energy counter, so a delta it reads is a
|
||
/// measurement. It sums exactly the packages the probe watched.
|
||
pub struct LinuxRaplCounter {
|
||
root: PathBuf,
|
||
packages: Vec<Package>,
|
||
running: Mutex<Running>,
|
||
}
|
||
|
||
impl LinuxRaplCounter {
|
||
/// This host's RAPL package counters, on a host where every one is
|
||
/// live. `Err(NoCounter)` when there is no package zone, or one is
|
||
/// unreadable or does not keep advancing under load;
|
||
/// [`crate::rapl_status`] says which.
|
||
pub fn try_detect() -> Result<Self> {
|
||
Self::try_detect_given(crate::rapl_status())
|
||
}
|
||
|
||
/// The constructor behind [`Self::try_detect`], given this host's
|
||
/// status: every status the gate does not admit declines. The counter
|
||
/// sums the package zones the host's probe watched.
|
||
pub(crate) fn try_detect_given(status: CounterStatus) -> Result<Self> {
|
||
Self::admitted(status, Path::new(POWERCAP_ROOT), host_packages)
|
||
}
|
||
|
||
/// The counter over `packages` under the powercap directory `root`,
|
||
/// when `status` admits it; `packages` is asked for only then.
|
||
pub(crate) fn admitted(status: CounterStatus, root: &Path, packages: impl FnOnce() -> Vec<Package>) -> Result<Self> {
|
||
if rail_liveness::admits_vendor_counter(status) {
|
||
Ok(Self::over(root, packages()))
|
||
} else {
|
||
Err(Error::NoCounter)
|
||
}
|
||
}
|
||
|
||
/// The counter over `packages` under `root`, ungated.
|
||
pub(crate) fn over(root: &Path, packages: Vec<Package>) -> Self {
|
||
Self { root: root.to_path_buf(), packages, running: Mutex::new(Running::Unread) }
|
||
}
|
||
}
|
||
|
||
impl JouleCounter for LinuxRaplCounter {
|
||
/// The running total. Fails — no figure, never a partial one — when
|
||
/// there is no package, when any package's counter cannot be read, or
|
||
/// once a package's energy between two reads is unknown. The counters
|
||
/// are read under the lock, so two threads' reads are folded in the
|
||
/// order they were taken, and an older read is never taken for a wrap.
|
||
fn read_microjoules(&self) -> Result<u64> {
|
||
if self.packages.is_empty() {
|
||
return Err(Error::NoCounter);
|
||
}
|
||
let mut running = self.running.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||
let now = read_packages(&self.root, &self.packages).ok_or_else(|| {
|
||
Error::Io(format!("a RAPL package zone's energy_uj under {} could not be read", self.root.display()))
|
||
})?;
|
||
running.advance(&self.packages, &now)
|
||
}
|
||
fn name(&self) -> &'static str {
|
||
"rapl"
|
||
}
|
||
}
|
||
|
||
/// The name a package's counter has in a liveness sample: its zone,
|
||
/// `intel-rapl:<index>`.
|
||
pub(crate) fn rail_name(index: u32) -> String {
|
||
format!("intel-rapl:{index}")
|
||
}
|
||
|
||
/// One liveness sample of the package counters `read` reads, in
|
||
/// `packages` order: each package's raw `energy_uj` as its own rail, under
|
||
/// its zone's name, so the rule asks EVERY package to keep advancing (the
|
||
/// figure sums them all). `None` for a failed read, which the probe
|
||
/// skips. A wrap shows as a fall, which is no advance.
|
||
pub(crate) fn sample(packages: &[Package], read: impl Fn() -> Option<Vec<u64>>) -> Option<Vec<(String, i64)>> {
|
||
let raw = read()?;
|
||
if raw.len() != packages.len() {
|
||
return None;
|
||
}
|
||
Some(packages.iter().zip(raw).map(|(p, uj)| (rail_name(p.index), i64::try_from(uj).unwrap_or(i64::MAX))).collect())
|
||
}
|
||
|
||
/// The RAPL probe of `packages` on a given reader. With no package it is
|
||
/// `Unavailable`, with nothing read and nothing burned. Otherwise it opens
|
||
/// when every package counter reads, and is then sampled by
|
||
/// [`rail_liveness::probe_counter`], with `tick` between samples, by the
|
||
/// clock `now`: live only when every package counter keeps advancing.
|
||
pub(crate) fn status_with(
|
||
packages: &[Package],
|
||
read: impl Fn() -> Option<Vec<u64>>,
|
||
tick: impl FnMut(),
|
||
now: impl Fn() -> Duration,
|
||
) -> CounterStatus {
|
||
if packages.is_empty() {
|
||
return CounterStatus::Unavailable;
|
||
}
|
||
rail_liveness::counter_status(
|
||
rail_liveness::probe_counter(
|
||
|| read().map(|_| ()),
|
||
|_| sample(packages, &read),
|
||
tick,
|
||
now,
|
||
crate::VENDOR_COUNTER_LIVENESS_BUDGET,
|
||
)
|
||
.as_ref(),
|
||
)
|
||
}
|
||
|
||
/// What one probe found under a powercap directory: its package zones,
|
||
/// and whether every one of their counters is live.
|
||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||
pub(crate) struct Host {
|
||
pub(crate) packages: Vec<Package>,
|
||
pub(crate) status: CounterStatus,
|
||
}
|
||
|
||
/// One probe of the package zones under `root`: list them
|
||
/// ([`packages_under`]), then probe all of their counters together
|
||
/// ([`status_with`]).
|
||
pub(crate) fn probe_root(root: &Path, tick: impl FnMut(), now: impl Fn() -> Duration) -> Host {
|
||
let packages = packages_under(root);
|
||
let status = status_with(&packages, || read_packages(root, &packages), tick, now);
|
||
Host { packages, status }
|
||
}
|
||
|
||
/// This host's package zones and their liveness, probed once per process
|
||
/// and cached: the real powercap directory, the CPU-burning tick (a
|
||
/// package counter must advance while this thread burns CPU) and the
|
||
/// monotonic clock. Behind [`crate::rapl_status`] and the counter's
|
||
/// constructor, so the counter sums exactly the packages the probe found
|
||
/// live.
|
||
#[cfg(target_os = "linux")]
|
||
pub(crate) fn host() -> &'static Host {
|
||
static HOST: std::sync::OnceLock<Host> = std::sync::OnceLock::new();
|
||
HOST.get_or_init(|| probe_root(Path::new(POWERCAP_ROOT), crate::clocks::cpu_burn_tick, crate::clocks::wall_now))
|
||
}
|
||
|
||
/// The package zones the host's probe watched; none off Linux.
|
||
fn host_packages() -> Vec<Package> {
|
||
#[cfg(target_os = "linux")]
|
||
{
|
||
host().packages.clone()
|
||
}
|
||
#[cfg(not(target_os = "linux"))]
|
||
{
|
||
Vec::new()
|
||
}
|
||
}
|
||
|
||
#[cfg(test)]
|
||
mod tests {
|
||
use super::*;
|
||
use std::cell::Cell;
|
||
|
||
/// A package zone's range as the kernel commonly reports it (2^32
|
||
/// units of 61.035 µJ), and a smaller one, so that a wrap taken at the
|
||
/// wrong zone's range shows.
|
||
const RANGE: u64 = 262_143_328_850;
|
||
const RANGE_B: u64 = 65_532_610_987;
|
||
|
||
/// A fake powercap directory for one test, removed when dropped.
|
||
struct Powercap(PathBuf);
|
||
|
||
impl Powercap {
|
||
fn new(tag: &str) -> Self {
|
||
let root = std::env::temp_dir().join(format!("eoc-meter-powercap-{tag}-{}", std::process::id()));
|
||
let _ = std::fs::remove_dir_all(&root);
|
||
std::fs::create_dir_all(&root).expect("create a fake powercap directory");
|
||
Self(root)
|
||
}
|
||
|
||
/// Add the zone directory `dir` (`intel-rapl:0`, `intel-rapl:0:1`,
|
||
/// `intel-rapl-mmio:0`, …) with its `name`, its `energy_uj` and,
|
||
/// when given, its `max_energy_range_uj`, newline-terminated as
|
||
/// the kernel writes them.
|
||
fn zone(&self, dir: &str, name: &str, energy_uj: u64, max: Option<u64>) -> &Self {
|
||
let d = self.0.join(dir);
|
||
std::fs::create_dir_all(&d).expect("create a fake zone");
|
||
std::fs::write(d.join("name"), format!("{name}\n")).expect("write name");
|
||
if let Some(m) = max {
|
||
std::fs::write(d.join("max_energy_range_uj"), format!("{m}\n")).expect("write max_energy_range_uj");
|
||
}
|
||
self.set(dir, energy_uj);
|
||
self
|
||
}
|
||
|
||
/// Set zone `dir`'s `energy_uj`.
|
||
fn set(&self, dir: &str, energy_uj: u64) {
|
||
std::fs::write(self.0.join(dir).join("energy_uj"), format!("{energy_uj}\n")).expect("write energy_uj");
|
||
}
|
||
|
||
fn root(&self) -> &Path {
|
||
&self.0
|
||
}
|
||
}
|
||
|
||
impl Drop for Powercap {
|
||
fn drop(&mut self) {
|
||
let _ = std::fs::remove_dir_all(&self.0);
|
||
}
|
||
}
|
||
|
||
/// A two-socket server's powercap directory as the kernel lists it:
|
||
/// the control type, two packages each with core or dram sub-zones,
|
||
/// a platform (`psys`) zone, a top-level `dram` zone, and the MMIO
|
||
/// interface republishing package 0.
|
||
fn two_socket_server(tag: &str) -> Powercap {
|
||
let pc = Powercap::new(tag);
|
||
std::fs::create_dir_all(pc.root().join("intel-rapl")).expect("the control type");
|
||
pc.zone("intel-rapl:0", "package-0", 1_000_000, Some(RANGE))
|
||
.zone("intel-rapl:0:0", "core", 400_000, Some(RANGE))
|
||
.zone("intel-rapl:0:1", "dram", 90_000, Some(RANGE_B))
|
||
.zone("intel-rapl:1", "package-1", 5_000_000, Some(RANGE_B))
|
||
.zone("intel-rapl:1:0", "dram", 70_000, Some(RANGE_B))
|
||
.zone("intel-rapl:2", "psys", 80_000_000, Some(RANGE))
|
||
.zone("intel-rapl:3", "dram", 3_000_000, Some(RANGE_B))
|
||
.zone("intel-rapl-mmio:0", "package-0", 1_000_000, Some(RANGE));
|
||
pc
|
||
}
|
||
|
||
fn pkg(index: u32, max: u64) -> Package {
|
||
Package { index, max_energy_range_uj: Some(max) }
|
||
}
|
||
|
||
/// The probe over a fake directory, run as the host's probe runs
|
||
/// (list, read, the rule), with a tick that sets each zone `step(n)`
|
||
/// names at the n-th tick, and a clock that moves 25 ms a tick.
|
||
fn probe_tree(pc: &Powercap, step: impl Fn(u64) -> Vec<(&'static str, u64)>) -> Host {
|
||
let ticks = Cell::new(0u32);
|
||
probe_root(
|
||
pc.root(),
|
||
|| {
|
||
ticks.set(ticks.get() + 1);
|
||
for (dir, uj) in step(u64::from(ticks.get())) {
|
||
pc.set(dir, uj);
|
||
}
|
||
},
|
||
|| Duration::from_millis(25) * ticks.get(),
|
||
)
|
||
}
|
||
|
||
/// A package zone is named `package-<n>`; `psys`, `dram`, `core` and
|
||
/// malformed names are not. A top-level zone is `intel-rapl:<n>`; a
|
||
/// sub-zone, the control type and the MMIO interface are not.
|
||
#[test]
|
||
fn only_a_top_level_zone_named_package_n_is_a_package() {
|
||
for name in ["package-0", "package-1\n", "package-12"] {
|
||
assert!(is_package_name(name), "{name:?}");
|
||
}
|
||
for name in ["psys", "psys\n", "dram", "core", "uncore", "package-", "package-x", "package-0a", "Package-0", ""] {
|
||
assert!(!is_package_name(name), "{name:?}");
|
||
}
|
||
assert_eq!(top_level_zone("intel-rapl:0"), Some(0));
|
||
assert_eq!(top_level_zone("intel-rapl:12"), Some(12));
|
||
for dir in ["intel-rapl", "intel-rapl:", "intel-rapl:0:0", "intel-rapl:1:2", "intel-rapl-mmio:0", "intel-rapl:x", "dtpm"] {
|
||
assert_eq!(top_level_zone(dir), None, "{dir:?}");
|
||
}
|
||
}
|
||
|
||
/// A two-socket server: the figure is both packages' energy and
|
||
/// nothing else — not `psys` (which includes the packages, so it would
|
||
/// count them twice), not a `dram` zone at the top level or under a
|
||
/// package, not a core sub-zone, not the MMIO copy of package 0.
|
||
#[test]
|
||
fn a_two_socket_server_sums_its_two_packages_and_nothing_else() {
|
||
let pc = two_socket_server("two-socket");
|
||
let packages = packages_under(pc.root());
|
||
assert_eq!(packages, vec![pkg(0, RANGE), pkg(1, RANGE_B)]);
|
||
assert_eq!(read_packages(pc.root(), &packages), Some(vec![1_000_000, 5_000_000]));
|
||
let counter = LinuxRaplCounter::over(pc.root(), packages);
|
||
let before = counter.read_microjoules().expect("both packages read");
|
||
assert_eq!(before, 1_000_000 + 5_000_000);
|
||
for (dir, uj) in [
|
||
("intel-rapl:0", 1_250_000),
|
||
("intel-rapl:1", 5_400_000),
|
||
("intel-rapl:0:0", 600_000),
|
||
("intel-rapl:0:1", 190_000),
|
||
("intel-rapl:1:0", 170_000),
|
||
("intel-rapl:2", 99_000_000),
|
||
("intel-rapl:3", 3_900_000),
|
||
("intel-rapl-mmio:0", 1_250_000),
|
||
] {
|
||
pc.set(dir, uj);
|
||
}
|
||
let after = counter.read_microjoules().expect("both packages read");
|
||
assert_eq!(after - before, 250_000 + 400_000, "package 0's energy and package 1's, nothing else");
|
||
}
|
||
|
||
/// Zone 0 is the platform zone and the package is zone 1: zone 0 is
|
||
/// neither summed nor sampled, and the figure is zone 1's.
|
||
#[test]
|
||
fn a_zone_0_named_psys_is_left_out() {
|
||
let pc = Powercap::new("psys-at-0");
|
||
pc.zone("intel-rapl:0", "psys", 70_000_000, Some(RANGE))
|
||
.zone("intel-rapl:1", "package-0", 2_000_000, Some(RANGE))
|
||
.zone("intel-rapl:1:0", "core", 1_000, Some(RANGE));
|
||
let packages = packages_under(pc.root());
|
||
assert_eq!(packages, vec![pkg(1, RANGE)]);
|
||
assert_eq!(
|
||
sample(&packages, || read_packages(pc.root(), &packages)),
|
||
Some(vec![("intel-rapl:1".to_string(), 2_000_000)]),
|
||
"the liveness sample holds the package alone"
|
||
);
|
||
let counter = LinuxRaplCounter::over(pc.root(), packages);
|
||
let before = counter.read_microjoules().expect("the package reads");
|
||
pc.set("intel-rapl:0", 90_000_000);
|
||
pc.set("intel-rapl:1", 2_300_000);
|
||
assert_eq!(counter.read_microjoules().expect("the package reads") - before, 300_000);
|
||
}
|
||
|
||
/// A `psys` zone 0 that moves at every tick beside a package that
|
||
/// holds: frozen. The platform zone is not the package, and is never
|
||
/// watched.
|
||
#[test]
|
||
fn a_moving_psys_at_zone_0_never_makes_a_held_package_live() {
|
||
let pc = Powercap::new("psys-moving");
|
||
pc.zone("intel-rapl:0", "psys", 70_000_000, Some(RANGE)).zone("intel-rapl:1", "package-0", 2_000_000, Some(RANGE));
|
||
let host = probe_tree(&pc, |t| vec![("intel-rapl:0", 70_000_000 + 99_000 * t)]);
|
||
assert_eq!(host, Host { packages: vec![pkg(1, RANGE)], status: CounterStatus::Frozen });
|
||
}
|
||
|
||
/// Every package must keep advancing, because the figure sums them
|
||
/// all: both moving at every tick is live; package 1 holding is
|
||
/// frozen, though package 0 and `psys` move at every tick.
|
||
#[test]
|
||
fn every_package_counter_must_be_live() {
|
||
let both = two_socket_server("live-both");
|
||
let host = probe_tree(&both, |t| vec![("intel-rapl:0", 1_000_000 + 40_000 * t), ("intel-rapl:1", 5_000_000 + 35_000 * t)]);
|
||
assert_eq!(host, Host { packages: vec![pkg(0, RANGE), pkg(1, RANGE_B)], status: CounterStatus::Live });
|
||
let one = two_socket_server("live-one");
|
||
let host = probe_tree(&one, |t| vec![("intel-rapl:0", 1_000_000 + 40_000 * t), ("intel-rapl:2", 80_000_000 + 99_000 * t)]);
|
||
assert_eq!(host.status, CounterStatus::Frozen, "package 1 held its value");
|
||
}
|
||
|
||
/// Each package wraps at its own range: package 0 rolls over at RANGE
|
||
/// while package 1 advances, then package 1 rolls over at RANGE_B. The
|
||
/// energy is each zone's delta modulo its own range, and the total
|
||
/// keeps rising across both wraps.
|
||
#[test]
|
||
fn each_package_wraps_at_its_own_range() {
|
||
let pc = Powercap::new("wrap");
|
||
pc.zone("intel-rapl:0", "package-0", RANGE - 1_000, Some(RANGE))
|
||
.zone("intel-rapl:1", "package-1", RANGE_B - 300, Some(RANGE_B));
|
||
let counter = LinuxRaplCounter::over(pc.root(), packages_under(pc.root()));
|
||
let r0 = counter.read_microjoules().expect("read");
|
||
pc.set("intel-rapl:0", 500);
|
||
pc.set("intel-rapl:1", RANGE_B - 100);
|
||
let r1 = counter.read_microjoules().expect("read across package 0's wrap");
|
||
assert_eq!(r1 - r0, (1_000 + 500) + 200);
|
||
pc.set("intel-rapl:0", 2_500);
|
||
pc.set("intel-rapl:1", 400);
|
||
let r2 = counter.read_microjoules().expect("read across package 1's wrap");
|
||
assert_eq!(r2 - r1, 2_000 + (100 + 400));
|
||
// At the other zone's range, package 1's wrap would be off by the
|
||
// difference of the two ranges.
|
||
assert_eq!(zone_delta_uj(RANGE_B - 100, 400, Some(RANGE_B)), Some(500));
|
||
assert_eq!(zone_delta_uj(RANGE_B - 100, 400, Some(RANGE)), Some(RANGE - RANGE_B + 500));
|
||
}
|
||
|
||
/// A package whose range is unknown (absent, or zero): summed while it
|
||
/// rises; once it goes down its energy since the last read is
|
||
/// unknown, so that read fails, and every later one too, since the
|
||
/// total would be short by that energy.
|
||
#[test]
|
||
fn a_fall_with_no_known_range_ends_the_figure() {
|
||
let pc = Powercap::new("no-range");
|
||
pc.zone("intel-rapl:0", "package-0", 10_000, None).zone("intel-rapl:1", "package-1", 20_000, Some(0));
|
||
let packages = packages_under(pc.root());
|
||
assert_eq!(packages, vec![Package { index: 0, max_energy_range_uj: None }, Package { index: 1, max_energy_range_uj: None }]);
|
||
let counter = LinuxRaplCounter::over(pc.root(), packages);
|
||
let r0 = counter.read_microjoules().expect("read");
|
||
pc.set("intel-rapl:0", 15_000);
|
||
pc.set("intel-rapl:1", 21_000);
|
||
assert_eq!(counter.read_microjoules().expect("read") - r0, 5_000 + 1_000);
|
||
pc.set("intel-rapl:0", 3_000);
|
||
assert!(matches!(counter.read_microjoules(), Err(Error::Backend(_))));
|
||
pc.set("intel-rapl:0", 4_000);
|
||
assert!(matches!(counter.read_microjoules(), Err(Error::Backend(_))), "a total short by the lost energy");
|
||
assert_eq!(zone_delta_uj(1_000, 10, Some(0)), None, "a zero range is no range");
|
||
}
|
||
|
||
/// A package that cannot be read fails the whole read: no figure that
|
||
/// leaves a package out.
|
||
#[test]
|
||
fn an_unreadable_package_fails_the_read() {
|
||
let pc = Powercap::new("unreadable");
|
||
pc.zone("intel-rapl:0", "package-0", 10_000, Some(RANGE)).zone("intel-rapl:1", "package-1", 20_000, Some(RANGE));
|
||
let packages = packages_under(pc.root());
|
||
let counter = LinuxRaplCounter::over(pc.root(), packages.clone());
|
||
std::fs::remove_file(pc.root().join("intel-rapl:1/energy_uj")).expect("remove");
|
||
assert_eq!(read_packages(pc.root(), &packages), None);
|
||
assert!(matches!(counter.read_microjoules(), Err(Error::Io(_))));
|
||
}
|
||
|
||
/// A host whose only zone is the platform (`psys`) has no package: the
|
||
/// probe reads and burns nothing and finds RAPL unavailable, and no
|
||
/// RAPL counter is constructed; one forced over no package never
|
||
/// reads. So too an empty powercap directory, and a missing one.
|
||
#[test]
|
||
fn a_host_with_only_psys_has_no_rapl_counter() {
|
||
let pc = Powercap::new("only-psys");
|
||
pc.zone("intel-rapl:0", "psys", 70_000_000, Some(RANGE));
|
||
let ticks = Cell::new(0u32);
|
||
let host = probe_root(pc.root(), || ticks.set(ticks.get() + 1), || Duration::from_millis(25) * ticks.get());
|
||
assert_eq!(host, Host { packages: Vec::new(), status: CounterStatus::Unavailable });
|
||
assert_eq!(ticks.get(), 0, "burned CPU probing a host with no package");
|
||
let asked = Cell::new(false);
|
||
let counter = LinuxRaplCounter::admitted(host.status, pc.root(), || {
|
||
asked.set(true);
|
||
host.packages.clone()
|
||
});
|
||
assert!(matches!(counter, Err(Error::NoCounter)));
|
||
assert!(!asked.get(), "listed the packages of a declined counter");
|
||
assert!(matches!(LinuxRaplCounter::over(pc.root(), Vec::new()).read_microjoules(), Err(Error::NoCounter)));
|
||
let empty = Powercap::new("empty");
|
||
assert_eq!(probe_root(empty.root(), || panic!("ticked"), || Duration::ZERO).status, CounterStatus::Unavailable);
|
||
let missing = Path::new("/no/such/powercap");
|
||
assert_eq!(probe_root(missing, || panic!("ticked"), || Duration::ZERO), Host { packages: Vec::new(), status: CounterStatus::Unavailable });
|
||
}
|
||
|
||
/// The RAPL probe on scripted reads of `packages`, with a clock that
|
||
/// only the tick moves, by 25 ms.
|
||
fn scripted(packages: &[Package], read: impl Fn(usize) -> Option<Vec<u64>>) -> CounterStatus {
|
||
let (clock, reads) = (Cell::new(Duration::ZERO), Cell::new(0usize));
|
||
status_with(
|
||
packages,
|
||
|| {
|
||
reads.set(reads.get() + 1);
|
||
read(reads.get() - 1)
|
||
},
|
||
|| clock.set(clock.get() + Duration::from_millis(25)),
|
||
|| clock.get(),
|
||
)
|
||
}
|
||
|
||
/// A package counter that advances at every read is live; one that
|
||
/// holds, steps once, or cannot be read (root-only `energy_uj`) is
|
||
/// declined, and so is one that reads only once. Of two packages,
|
||
/// both must advance.
|
||
#[test]
|
||
fn the_rapl_probe_admits_only_counters_that_keep_advancing() {
|
||
let one = [pkg(0, RANGE)];
|
||
assert_eq!(scripted(&one, |i| Some(vec![81_000_000 + 1_400_000 * i as u64])), CounterStatus::Live);
|
||
assert_eq!(scripted(&one, |_| Some(vec![81_000_000])), CounterStatus::Frozen, "a held value");
|
||
assert_eq!(scripted(&one, |i| Some(vec![if i < 9 { 1 } else { 2 }])), CounterStatus::Frozen, "one step");
|
||
assert_eq!(scripted(&one, |_| None), CounterStatus::Unavailable, "EACCES on every read");
|
||
assert_eq!(scripted(&one, |i| (i == 0).then(|| vec![5])), CounterStatus::Unavailable, "read once, then never");
|
||
let two = [pkg(0, RANGE), pkg(1, RANGE)];
|
||
assert_eq!(scripted(&two, |i| Some(vec![1_000 + 7 * i as u64, 9_000 + 5 * i as u64])), CounterStatus::Live);
|
||
assert_eq!(scripted(&two, |i| Some(vec![1_000 + 7 * i as u64, 9_000])), CounterStatus::Frozen, "package 1 held");
|
||
assert_eq!(scripted(&two, |i| Some(vec![1_000, 9_000 + 5 * i as u64])), CounterStatus::Frozen, "package 0 held");
|
||
assert_eq!(scripted(&[], |_| panic!("read a host with no package")), CounterStatus::Unavailable);
|
||
}
|
||
|
||
/// The sampler hands each package's raw counter as its own rail, and
|
||
/// saturates rather than wrapping a value past `i64::MAX`.
|
||
#[test]
|
||
fn the_rapl_sample_is_each_package_counter_raw() {
|
||
let two = [pkg(0, RANGE), pkg(3, RANGE)];
|
||
assert_eq!(
|
||
sample(&two, || Some(vec![42, u64::MAX])),
|
||
Some(vec![("intel-rapl:0".to_string(), 42), ("intel-rapl:3".to_string(), i64::MAX)])
|
||
);
|
||
assert_eq!(sample(&two, || None), None);
|
||
assert_eq!(sample(&two, || Some(vec![42])), None, "a read that missed a package");
|
||
assert_eq!(read_u64(Path::new("/no/such/powercap/energy_uj")), None);
|
||
}
|
||
|
||
/// Only a live status constructs the counter.
|
||
#[test]
|
||
fn only_a_live_rapl_status_constructs_the_counter() {
|
||
assert!(LinuxRaplCounter::try_detect_given(CounterStatus::Live).is_ok());
|
||
for status in [CounterStatus::Unavailable, CounterStatus::Frozen] {
|
||
assert!(
|
||
matches!(LinuxRaplCounter::try_detect_given(status), Err(Error::NoCounter)),
|
||
"{status:?} constructed the RAPL counter"
|
||
);
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
#[cfg(target_os = "linux")]
|
||
pub use linux_rapl::LinuxRaplCounter;
|
||
|
||
#[cfg(target_os = "macos")]
|
||
mod macos_powermetrics {
|
||
use super::*;
|
||
|
||
/// Stands in for `powermetrics(1)`. It is detected whenever the binary
|
||
/// is on the PATH, but it does not run or parse it (that needs root):
|
||
/// every read returns `Err(NoCounter)`, so callers keep their own
|
||
/// estimate. It never degrades to `StubCounter`.
|
||
pub struct MacosPowermetricsCounter;
|
||
|
||
impl MacosPowermetricsCounter {
|
||
/// Attempt to detect `powermetrics` availability.
|
||
pub fn try_detect() -> Result<Self> {
|
||
// Only the binary's presence on the PATH is checked; reads
|
||
// always fail (see `read_microjoules`).
|
||
let ok = std::process::Command::new("which")
|
||
.arg("powermetrics")
|
||
.output()
|
||
.map(|o| o.status.success())
|
||
.unwrap_or(false);
|
||
if ok { Ok(Self) } else { Err(Error::NoCounter) }
|
||
}
|
||
}
|
||
|
||
impl JouleCounter for MacosPowermetricsCounter {
|
||
fn read_microjoules(&self) -> Result<u64> {
|
||
// Real implementation would parse `powermetrics --samplers
|
||
// cpu_power -i 100 -n 1 --hide-cpu-duty-cycle`. For the
|
||
// reference implementation we always error, so no consumer
|
||
// gets a figure from it and each keeps its own estimate.
|
||
Err(Error::NoCounter)
|
||
}
|
||
fn name(&self) -> &'static str {
|
||
"powermetrics"
|
||
}
|
||
}
|
||
}
|
||
|
||
#[cfg(target_os = "macos")]
|
||
pub use macos_powermetrics::MacosPowermetricsCounter;
|
||
|
||
// When a process may read its own energy count, kept apart from the FFI so
|
||
// the rule and the one byte layout it depends on run under test on every
|
||
// platform, with scripted counts and scripted sleeps.
|
||
//
|
||
// The kernel adds a thread's CPU energy to its process's count
|
||
// (`ri_energy_nj`) only when the thread leaves a core: a context switch, or
|
||
// the end of its quantum, about 10 ms of its CPU. A process's read of its own
|
||
// count does not take the reading thread off its core — its CPU time is
|
||
// current at the read, its energy is not. So the count read from a thread
|
||
// that has been running lacks what that thread spent since it last left a
|
||
// core. Bracket a span with two such reads and the first puts up to a
|
||
// quantum of energy from BEFORE the span inside it, and the second leaves
|
||
// the span's tail out. Measured on an M3 Ultra, macOS 27.0.1 (load average
|
||
// about 14, three runs of 40 spans each, against the tests' own reader that
|
||
// sleeps before each of its reads): of spans over 1 ms of the reading
|
||
// thread's CPU, 39, 40 and 40 read nothing or under two thirds of the work;
|
||
// of spans over a 2 ms sleep opened straight after 8 ms of that thread's
|
||
// CPU, 40, 40 and 37 read the burn before them as the sleep's energy, about
|
||
// 41 mJ for a sleep.
|
||
//
|
||
// A thread that sleeps leaves its core — unless the sleep's timer fires
|
||
// before the thread has blocked, and then it goes on without leaving. So the
|
||
// reading thread sleeps and CHECKS: its own energy count
|
||
// (`proc_pidinfo`'s `PROC_PIDTHREADCOUNTS`) is moved by the kernel only when
|
||
// it adds the thread's energy to the process's, so a rise across a sleep is
|
||
// the thread having left its core after everything it did before the sleep.
|
||
// The process's context-switch count is no such check, since any other
|
||
// thread's switch moves it. The sleeps grow, so one that did not block is
|
||
// followed by one that much more surely does.
|
||
#[cfg_attr(not(target_os = "macos"), allow(dead_code))]
|
||
mod reading_thread {
|
||
use std::time::Duration;
|
||
|
||
/// The sleeps the reading thread takes, in turn, until one is seen to
|
||
/// take it off its core: 6.25 ms in all on a read that never is.
|
||
pub(crate) const SLEEPS: [Duration; 4] =
|
||
[Duration::from_micros(50), Duration::from_micros(200), Duration::from_millis(1), Duration::from_millis(5)];
|
||
|
||
/// Whether the reading thread was seen to leave its core: its own energy
|
||
/// count, read by `count`, rose across one of `sleeps`, taken in turn
|
||
/// through `sleep`. A count that held across every sleep, or fell, is no
|
||
/// evidence that it left, and a count that cannot be read ends the wait
|
||
/// at once: `false`, and the process's count is not read on it.
|
||
pub(crate) fn left_its_core(
|
||
sleeps: &[Duration],
|
||
mut count: impl FnMut() -> Option<u64>,
|
||
mut sleep: impl FnMut(Duration),
|
||
) -> bool {
|
||
for &pause in sleeps {
|
||
let Some(before) = count() else { return false };
|
||
sleep(pause);
|
||
match count() {
|
||
Some(after) if after > before => return true,
|
||
Some(_) => {}
|
||
None => return false,
|
||
}
|
||
}
|
||
false
|
||
}
|
||
|
||
/// `PROC_PIDTHREADCOUNTS`: the `proc_pidinfo` flavor that returns one
|
||
/// thread's counts, asked by its 64-bit thread id. It is in the kernel's
|
||
/// `<sys/proc_info_private.h>` (macOS 13 and later), not the SDK's
|
||
/// public header, so [`energy_nj`] reads its result in one layout only.
|
||
pub(crate) const THREAD_COUNTS_FLAVOR: i32 = 34;
|
||
|
||
/// The size of the buffer a read of the thread's counts hands the
|
||
/// kernel: room for the header and twelve performance levels.
|
||
pub(crate) const THREAD_COUNTS_SIZE: usize = 8 + 12 * 40;
|
||
|
||
/// The thread's CPU energy in nanojoules, summed over the CPU performance
|
||
/// levels, from the `wrote` bytes the call says it wrote into `counts`.
|
||
/// The kernel writes a `uint16_t` count of performance levels and six
|
||
/// bytes of padding, then for each level five `uint64_t`: instructions,
|
||
/// cycles, user time, system time, energy (nJ).
|
||
///
|
||
/// `None` unless the call wrote exactly that: the 8-byte header and one
|
||
/// 40-byte record for each level the header names, at least one. A
|
||
/// failed call and a result of any other size — a layout this crate does
|
||
/// not know — are no count, never a field read from the wrong place.
|
||
/// Seen on an M5 Max and an M3 Ultra (macOS 27.0.1): 88 bytes, two
|
||
/// levels.
|
||
pub(crate) fn energy_nj(counts: &[u8], wrote: i32) -> Option<u64> {
|
||
let levels = usize::from(u16::from_ne_bytes([*counts.first()?, *counts.get(1)?]));
|
||
let size = 8 + levels * 40;
|
||
if levels == 0 || size > counts.len() || usize::try_from(wrote).ok() != Some(size) {
|
||
return None;
|
||
}
|
||
let mut nj = 0u64;
|
||
for level in 0..levels {
|
||
let at = 8 + level * 40 + 32;
|
||
let mut energy = [0u8; 8];
|
||
energy.copy_from_slice(&counts[at..at + 8]);
|
||
nj = nj.saturating_add(u64::from_ne_bytes(energy));
|
||
}
|
||
Some(nj)
|
||
}
|
||
|
||
#[cfg(test)]
|
||
mod tests {
|
||
use super::*;
|
||
use std::cell::RefCell;
|
||
|
||
/// The thread is seen to leave its core only when its own count
|
||
/// rose across a sleep: the sleeps are taken in turn until one does.
|
||
/// A count that held, fell or could not be read is not a thread
|
||
/// seen to leave, and an unreadable count takes no further sleep.
|
||
#[test]
|
||
fn a_thread_is_seen_to_leave_its_core_only_by_its_own_count_rising() {
|
||
let sleeps = [Duration::from_micros(50), Duration::from_micros(200), Duration::from_millis(1)];
|
||
let seen = |counts: &[Option<u64>]| {
|
||
let mut counts = counts.iter().copied();
|
||
let slept = RefCell::new(Vec::new());
|
||
let left = left_its_core(&sleeps, || counts.next().flatten(), |pause| slept.borrow_mut().push(pause));
|
||
(left, slept.into_inner())
|
||
};
|
||
assert_eq!(seen(&[Some(300), Some(301)]), (true, vec![sleeps[0]]), "rose across the first sleep");
|
||
assert_eq!(
|
||
seen(&[Some(300), Some(300), Some(300), Some(300), Some(300), Some(9_000)]),
|
||
(true, sleeps.to_vec()),
|
||
"held across two sleeps, rose across the third"
|
||
);
|
||
assert_eq!(seen(&[Some(300); 6]), (false, sleeps.to_vec()), "held across every sleep");
|
||
assert_eq!(seen(&[Some(300), Some(200), Some(200), Some(100), Some(100), Some(50)]), (false, sleeps.to_vec()), "fell");
|
||
assert_eq!(seen(&[None]), (false, vec![]), "unreadable before any sleep");
|
||
assert_eq!(seen(&[Some(300), None]), (false, vec![sleeps[0]]), "unreadable after a sleep");
|
||
assert!(!left_its_core(&[], || Some(1), |_| panic!("slept")), "no sleep, nothing seen");
|
||
assert_eq!(SLEEPS.iter().sum::<Duration>(), Duration::from_micros(6_250));
|
||
assert!(SLEEPS.windows(2).all(|w| w[0] < w[1]) && SLEEPS[0] >= Duration::from_micros(50), "{SLEEPS:?}");
|
||
}
|
||
|
||
/// A thread's counts as the kernel writes them: `levels` in the
|
||
/// first two bytes, six of padding, then five u64 a level, the fifth
|
||
/// its energy.
|
||
fn counts(levels: u16, energy_nj: &[u64]) -> Vec<u8> {
|
||
let mut bytes = vec![0u8; THREAD_COUNTS_SIZE];
|
||
bytes[..2].copy_from_slice(&levels.to_ne_bytes());
|
||
for (level, nj) in energy_nj.iter().enumerate() {
|
||
for (field, word) in [9_001u64, 9_002, 9_003, 9_004, *nj].into_iter().enumerate() {
|
||
let at = 8 + level * 40 + field * 8;
|
||
bytes[at..at + 8].copy_from_slice(&word.to_ne_bytes());
|
||
}
|
||
}
|
||
bytes
|
||
}
|
||
|
||
/// The energy is each level's fifth u64, summed; and a result of any
|
||
/// size but the header and one record per level named is no count.
|
||
#[test]
|
||
fn the_threads_energy_is_read_only_from_the_layout_known() {
|
||
assert_eq!(energy_nj(&counts(2, &[4_000_000, 600]), 88), Some(4_000_600), "two levels, 88 bytes");
|
||
assert_eq!(energy_nj(&counts(1, &[42]), 48), Some(42));
|
||
assert_eq!(energy_nj(&counts(3, &[1, 10, 100]), 128), Some(111));
|
||
assert_eq!(energy_nj(&counts(2, &[u64::MAX, 1]), 88), Some(u64::MAX), "the sum saturates");
|
||
let two = counts(2, &[4_000_000, 600]);
|
||
for (wrote, why) in [(-1, "the call failed"), (0, "nothing written"), (48, "one record"), (87, "short"), (128, "long")] {
|
||
assert_eq!(energy_nj(&two, wrote), None, "{why}");
|
||
}
|
||
assert_eq!(energy_nj(&counts(0, &[]), 8), None, "no performance level");
|
||
assert_eq!(energy_nj(&counts(13, &[]), 8 + 13 * 40), None, "more levels than the buffer holds");
|
||
assert_eq!(energy_nj(&[2], 88), None, "no header");
|
||
assert_eq!(THREAD_COUNTS_FLAVOR, 34);
|
||
}
|
||
}
|
||
}
|
||
|
||
/// This process's own CPU energy on macOS, as the kernel attributes it to
|
||
/// this process's threads: `ri_energy_nj` of `proc_pid_rusage(getpid(),
|
||
/// RUSAGE_INFO_V6)` (libproc; public, no privilege, no entitlement).
|
||
///
|
||
/// The domain, exactly: the CPU energy of **this process's threads**, all of
|
||
/// them, cumulative since the process started. Not a child process's (a
|
||
/// child's energy is its own, and is not added to the parent's when it is
|
||
/// reaped), and not the GPU, the Neural Engine, DRAM, or anything else on the
|
||
/// SoC. So it measures work this process runs on the CPU, in-process, and
|
||
/// nothing else; and it measures all of that, including another thread's
|
||
/// unrelated work during the same span.
|
||
///
|
||
/// That is why [`crate::detect`] never selects it: eoc's counter brackets
|
||
/// stages that may run their work in a subprocess (a CLI-backed tool, a
|
||
/// scoring interpreter) or on the GPU, and a span over such work would read
|
||
/// only this process's share — a figure for work it did not see, tagged
|
||
/// measured. A caller whose metered work runs on the CPU in this process
|
||
/// opts in with [`crate::detect_in_process`] or
|
||
/// [`MacosProcessEnergyCounter::try_new`].
|
||
///
|
||
/// **Each read is taken only once the reading thread is seen to have left
|
||
/// its core** (`reading_thread`): the kernel counts a thread's energy into
|
||
/// the process's only then, so a read from a thread still on its core lacks
|
||
/// what that thread spent since it last left one, and a span bracketed by
|
||
/// two such reads holds what the thread spent before the span and lacks its
|
||
/// own tail. The read sleeps 50 µs, then 200 µs, 1 ms and 5 ms, until the
|
||
/// reading thread's own energy count rose across a sleep, and only then
|
||
/// reads the process. A read whose thread was not seen to leave is an
|
||
/// error, not a short count: every consumer here takes a failed read as no
|
||
/// span, and keeps its estimate. With it, the same spans on the same host,
|
||
/// quiet and under 40 busy loops (110 runs of 40 spans of each kind): every
|
||
/// 1 ms span read 79% or more of the tests' own window around it (98% on
|
||
/// the quiet host), and no sleep read more than 3% of the window that held
|
||
/// the burn before it.
|
||
///
|
||
/// What that costs, the sleep and the wait for a core after it: 83 µs a
|
||
/// read at the median on the quiet host and 113 µs at most; under 40 busy
|
||
/// loops (load average up to 105 on 32 cores), 226 µs at the median over
|
||
/// the runs, 1.5 ms at the median of the worst run, and up to 38 ms for one
|
||
/// read; against under 1 µs for the read alone. A read that fails costs all
|
||
/// four sleeps (6.25 ms) and their waits. A span also holds its closing
|
||
/// read's own wake-up, a few tens of microjoules (31 µJ for a span over a
|
||
/// 500 ms sleep), which is most of the figure for a span over much less
|
||
/// than a millisecond of work: such a span is a true reading of what this
|
||
/// process spent between its two reads, over that work and not under it. A
|
||
/// caller with many short pieces of work brackets the lot with one span,
|
||
/// not each with its own.
|
||
///
|
||
/// Only the reading thread is seen to leave. Another thread of the process
|
||
/// that is on a core at a read is short by what it spent since it last left
|
||
/// one, up to a quantum: work on the thread that takes both reads is read
|
||
/// whole, and work on other threads (a thread pool, or a task an async
|
||
/// runtime moved to another thread between the two reads) to within a
|
||
/// quantum for each such thread at each end. The liveness probe samples the
|
||
/// count with no sleep: it asks only whether the count keeps advancing under
|
||
/// a burn, which it does at every quantum.
|
||
#[cfg(target_os = "macos")]
|
||
#[allow(unsafe_code)]
|
||
mod macos_process_energy {
|
||
use super::*;
|
||
use crate::{rail_liveness, reading_thread};
|
||
use std::os::raw::{c_int, c_void};
|
||
|
||
/// `RUSAGE_INFO_V6` in `<sys/resource.h>`.
|
||
const RUSAGE_INFO_V6: c_int = 6;
|
||
|
||
/// `struct rusage_info_v6` from the macOS 27 SDK's `<sys/resource.h>`,
|
||
/// field for field. Only `ri_energy_nj` is read; the layout up to it must
|
||
/// match exactly, and the size must cover what the kernel writes.
|
||
#[repr(C)]
|
||
#[derive(Default)]
|
||
struct RusageInfoV6 {
|
||
ri_uuid: [u8; 16],
|
||
ri_user_time: u64,
|
||
ri_system_time: u64,
|
||
ri_pkg_idle_wkups: u64,
|
||
ri_interrupt_wkups: u64,
|
||
ri_pageins: u64,
|
||
ri_wired_size: u64,
|
||
ri_resident_size: u64,
|
||
ri_phys_footprint: u64,
|
||
ri_proc_start_abstime: u64,
|
||
ri_proc_exit_abstime: u64,
|
||
ri_child_user_time: u64,
|
||
ri_child_system_time: u64,
|
||
ri_child_pkg_idle_wkups: u64,
|
||
ri_child_interrupt_wkups: u64,
|
||
ri_child_pageins: u64,
|
||
ri_child_elapsed_abstime: u64,
|
||
ri_diskio_bytesread: u64,
|
||
ri_diskio_byteswritten: u64,
|
||
ri_cpu_time_qos_default: u64,
|
||
ri_cpu_time_qos_maintenance: u64,
|
||
ri_cpu_time_qos_background: u64,
|
||
ri_cpu_time_qos_utility: u64,
|
||
ri_cpu_time_qos_legacy: u64,
|
||
ri_cpu_time_qos_user_initiated: u64,
|
||
ri_cpu_time_qos_user_interactive: u64,
|
||
ri_billed_system_time: u64,
|
||
ri_serviced_system_time: u64,
|
||
ri_logical_writes: u64,
|
||
ri_lifetime_max_phys_footprint: u64,
|
||
ri_instructions: u64,
|
||
ri_cycles: u64,
|
||
ri_billed_energy: u64,
|
||
ri_serviced_energy: u64,
|
||
ri_interval_max_phys_footprint: u64,
|
||
ri_runnable_time: u64,
|
||
ri_flags: u64,
|
||
ri_user_ptime: u64,
|
||
ri_system_ptime: u64,
|
||
ri_pinstructions: u64,
|
||
ri_pcycles: u64,
|
||
ri_energy_nj: u64,
|
||
ri_penergy_nj: u64,
|
||
ri_secure_time_in_system: u64,
|
||
ri_secure_ptime_in_system: u64,
|
||
ri_neural_footprint: u64,
|
||
ri_lifetime_max_neural_footprint: u64,
|
||
ri_interval_max_neural_footprint: u64,
|
||
ri_conclave_footprint: u64,
|
||
ri_page_wait_time_mach: u64,
|
||
ri_page_cache_hits: u64,
|
||
ri_reserved: [u64; 6],
|
||
}
|
||
|
||
// `sizeof(struct rusage_info_v6)` and `offsetof(.., ri_energy_nj)`, as a C
|
||
// program compiled against the macOS 27 SDK prints them.
|
||
const _: () = assert!(std::mem::size_of::<RusageInfoV6>() == 464);
|
||
const _: () = assert!(std::mem::offset_of!(RusageInfoV6, ri_energy_nj) == 336);
|
||
|
||
unsafe extern "C" {
|
||
fn proc_pid_rusage(pid: c_int, flavor: c_int, buffer: *mut c_void) -> c_int;
|
||
fn proc_pidinfo(pid: c_int, flavor: c_int, arg: u64, buffer: *mut c_void, buffersize: c_int) -> c_int;
|
||
fn pthread_self() -> *mut c_void;
|
||
fn pthread_threadid_np(thread: *mut c_void, thread_id: *mut u64) -> c_int;
|
||
fn getpid() -> c_int;
|
||
}
|
||
|
||
/// This process's id.
|
||
pub(crate) fn this_pid() -> i32 {
|
||
unsafe { getpid() }
|
||
}
|
||
|
||
/// The calling thread's own cumulative CPU energy, in nanojoules: the
|
||
/// count a read watches to see the thread leave its core. `None` if the
|
||
/// thread's id or its counts cannot be read, or the counts come back in
|
||
/// a layout [`reading_thread::energy_nj`] does not know.
|
||
pub(crate) fn this_thread_energy_nj() -> Option<u64> {
|
||
let mut thread = 0u64;
|
||
if unsafe { pthread_threadid_np(pthread_self(), &mut thread) } != 0 {
|
||
return None;
|
||
}
|
||
let mut counts = [0u8; reading_thread::THREAD_COUNTS_SIZE];
|
||
let size = c_int::try_from(counts.len()).ok()?;
|
||
let wrote = unsafe {
|
||
proc_pidinfo(this_pid(), reading_thread::THREAD_COUNTS_FLAVOR, thread, counts.as_mut_ptr() as *mut c_void, size)
|
||
};
|
||
reading_thread::energy_nj(&counts, wrote)
|
||
}
|
||
|
||
/// What a read of this process's own count says when the reading thread
|
||
/// was not seen to leave its core before it.
|
||
const NOT_SEEN_OFF_CORE: &str = "the reading thread was not seen to leave its core before a read of its own \
|
||
process's energy: the count could lack what the thread spent since it last \
|
||
did, so it was not read";
|
||
|
||
/// The cumulative CPU energy, in nanojoules, the kernel attributes to the
|
||
/// threads of process `pid`; `None` when `proc_pid_rusage` fails (no such
|
||
/// process, or one this user may not inspect).
|
||
pub(crate) fn energy_nj(pid: i32) -> Option<u64> {
|
||
let mut info = RusageInfoV6::default();
|
||
let rc = unsafe { proc_pid_rusage(pid, RUSAGE_INFO_V6, &mut info as *mut RusageInfoV6 as *mut c_void) };
|
||
(rc == 0).then_some(info.ri_energy_nj)
|
||
}
|
||
|
||
/// This process's own CPU energy counter; see the module documentation
|
||
/// for its domain. Constructed only where the counter is live
|
||
/// ([`crate::process_energy_status`]), and then a delta it reads is a
|
||
/// measurement of this process's CPU work over the span.
|
||
#[derive(Debug)]
|
||
pub struct MacosProcessEnergyCounter {
|
||
pid: i32,
|
||
/// The reading thread's own energy count, which every read is
|
||
/// checked against: [`this_thread_energy_nj`], but for a test of a
|
||
/// thread that is never seen to leave its core.
|
||
thread_count: fn() -> Option<u64>,
|
||
}
|
||
|
||
impl MacosProcessEnergyCounter {
|
||
/// This process's energy counter, on a host where it is live.
|
||
/// `Err(NoCounter)` otherwise; [`crate::process_energy_status`] says
|
||
/// why. For callers whose metered work runs on the CPU in this
|
||
/// process only: not a subprocess's work, not the GPU's.
|
||
pub fn try_new() -> Result<Self> {
|
||
Self::try_new_given(crate::process_energy_status())
|
||
}
|
||
|
||
/// The constructor behind [`Self::try_new`], given this host's status:
|
||
/// every status the gate does not admit declines.
|
||
pub(crate) fn try_new_given(status: CounterStatus) -> Result<Self> {
|
||
if rail_liveness::admits_vendor_counter(status) {
|
||
Ok(Self { pid: this_pid(), thread_count: this_thread_energy_nj })
|
||
} else {
|
||
Err(Error::NoCounter)
|
||
}
|
||
}
|
||
}
|
||
|
||
impl JouleCounter for MacosProcessEnergyCounter {
|
||
/// This process's cumulative CPU energy since it started, in µJ
|
||
/// (`ri_energy_nj` / 1000), read once the calling thread is seen to
|
||
/// have left its core, so the count holds all that thread spent
|
||
/// before the read. `Err` where it was not seen to leave within
|
||
/// [`reading_thread::SLEEPS`]: the process's count is then not read,
|
||
/// and a caller that brackets a span with this read has no reading
|
||
/// of the span, never a short one. Sleeps at least 50 µs.
|
||
fn read_microjoules(&self) -> Result<u64> {
|
||
if !reading_thread::left_its_core(&reading_thread::SLEEPS, self.thread_count, std::thread::sleep) {
|
||
return Err(Error::Backend(NOT_SEEN_OFF_CORE.to_string()));
|
||
}
|
||
energy_nj(self.pid)
|
||
.map(|nj| nj / 1_000)
|
||
.ok_or_else(|| Error::Backend("proc_pid_rusage(RUSAGE_INFO_V6) failed".to_string()))
|
||
}
|
||
fn name(&self) -> &'static str {
|
||
"process-energy"
|
||
}
|
||
}
|
||
|
||
/// One liveness sample of the counter `read` reads: its raw nanojoules
|
||
/// as the one rail. `None` for a failed read, which the probe skips.
|
||
fn sample(read: impl Fn() -> Option<u64>) -> Option<Vec<(String, i64)>> {
|
||
let nj = read()?;
|
||
Some(vec![("ri_energy_nj".to_string(), i64::try_from(nj).unwrap_or(i64::MAX))])
|
||
}
|
||
|
||
/// The process counter's probe on a given reader: it opens when the
|
||
/// counter reads at all, and is then sampled by
|
||
/// [`rail_liveness::probe_counter`], with `tick` between samples, by the
|
||
/// clock `now`.
|
||
pub(crate) fn status_with(
|
||
read: impl Fn() -> Option<u64>,
|
||
tick: impl FnMut(),
|
||
now: impl Fn() -> std::time::Duration,
|
||
) -> CounterStatus {
|
||
rail_liveness::counter_status(
|
||
rail_liveness::probe_counter(
|
||
|| read().map(|_| ()),
|
||
|_| sample(&read),
|
||
tick,
|
||
now,
|
||
crate::VENDOR_COUNTER_LIVENESS_BUDGET,
|
||
)
|
||
.as_ref(),
|
||
)
|
||
}
|
||
|
||
/// The real probe: this process's counter, the CPU-burning tick (this
|
||
/// thread's burn is this process's work, so a live counter advances in
|
||
/// every interval) and the monotonic clock. Cached by
|
||
/// [`crate::process_energy_status`], its only caller. The probe samples
|
||
/// the count with no sleep before a sample.
|
||
///
|
||
/// `Unavailable`, the probe not run, where the calling thread's own
|
||
/// energy count does not read: every read the counter takes is checked
|
||
/// against that count, so a counter there could never read.
|
||
pub(crate) fn host_status() -> CounterStatus {
|
||
let pid = this_pid();
|
||
status_where(this_thread_energy_nj().is_some(), || {
|
||
status_with(|| energy_nj(pid), crate::clocks::cpu_burn_tick, crate::clocks::wall_now)
|
||
})
|
||
}
|
||
|
||
/// The counter's status on a host where the reading thread's own energy
|
||
/// count reads, or does not: what `probe` finds where it does, and
|
||
/// `Unavailable`, with no probe run, where it does not.
|
||
pub(crate) fn status_where(thread_count_reads: bool, probe: impl FnOnce() -> CounterStatus) -> CounterStatus {
|
||
if thread_count_reads {
|
||
probe()
|
||
} else {
|
||
CounterStatus::Unavailable
|
||
}
|
||
}
|
||
|
||
#[cfg(test)]
|
||
mod tests {
|
||
use super::*;
|
||
use std::cell::Cell;
|
||
use std::time::Duration;
|
||
|
||
/// The probe on scripted reads, with a clock that only the tick
|
||
/// moves, by 25 ms.
|
||
fn scripted(read: impl Fn(usize) -> Option<u64>) -> CounterStatus {
|
||
let (clock, reads) = (Cell::new(Duration::ZERO), Cell::new(0usize));
|
||
status_with(
|
||
|| {
|
||
reads.set(reads.get() + 1);
|
||
read(reads.get() - 1)
|
||
},
|
||
|| clock.set(clock.get() + Duration::from_millis(25)),
|
||
|| clock.get(),
|
||
)
|
||
}
|
||
|
||
/// A counter that advances at every read is live; one that holds (a
|
||
/// constant), steps once, or cannot be read is declined.
|
||
#[test]
|
||
fn the_process_probe_admits_only_a_counter_that_keeps_advancing() {
|
||
assert_eq!(scripted(|i| Some(1_000_000 + 60_000_000 * i as u64)), CounterStatus::Live);
|
||
assert_eq!(scripted(|_| Some(1_000_000)), CounterStatus::Frozen, "a constant");
|
||
assert_eq!(scripted(|i| Some(if i < 9 { 1 } else { 2 })), CounterStatus::Frozen, "one step");
|
||
assert_eq!(scripted(|_| None), CounterStatus::Unavailable, "proc_pid_rusage failing");
|
||
}
|
||
|
||
/// Only a live status constructs the counter, and it reads this
|
||
/// process.
|
||
#[test]
|
||
fn only_a_live_status_constructs_the_process_counter() {
|
||
for status in [CounterStatus::Unavailable, CounterStatus::Frozen] {
|
||
assert!(
|
||
matches!(MacosProcessEnergyCounter::try_new_given(status), Err(Error::NoCounter)),
|
||
"{status:?} constructed the process counter"
|
||
);
|
||
}
|
||
let c = MacosProcessEnergyCounter::try_new_given(CounterStatus::Live).expect("Live constructs");
|
||
assert_eq!(c.pid, this_pid());
|
||
assert_eq!(c.name(), "process-energy");
|
||
}
|
||
|
||
/// A process that does not exist has no energy to read.
|
||
#[test]
|
||
fn no_process_no_reading() {
|
||
assert_eq!(energy_nj(-1), None);
|
||
assert!(MacosProcessEnergyCounter { pid: -1, thread_count: this_thread_energy_nj }.read_microjoules().is_err());
|
||
}
|
||
|
||
/// A read whose thread is never seen to leave its core is no read:
|
||
/// an error after every sleep, never the count as it stands — which
|
||
/// could lack what the thread spent — and an error at once when the
|
||
/// thread's own count cannot be read. A consumer takes either as no
|
||
/// span and keeps its estimate.
|
||
#[test]
|
||
fn a_read_whose_thread_is_never_seen_off_its_core_fails() {
|
||
let held = MacosProcessEnergyCounter { pid: this_pid(), thread_count: || Some(7) };
|
||
let started = std::time::Instant::now();
|
||
let read = held.read_microjoules();
|
||
let took = started.elapsed();
|
||
assert!(matches!(&read, Err(Error::Backend(why)) if why.contains("not seen to leave its core")), "{read:?}");
|
||
let every_sleep: Duration = reading_thread::SLEEPS.iter().sum();
|
||
assert!(took >= every_sleep, "gave up after {took:?}, before all of {:?}", reading_thread::SLEEPS);
|
||
|
||
let unreadable = MacosProcessEnergyCounter { pid: this_pid(), thread_count: || None };
|
||
let read = unreadable.read_microjoules();
|
||
assert!(matches!(&read, Err(Error::Backend(why)) if why.contains("not seen to leave its core")), "{read:?}");
|
||
}
|
||
|
||
/// On this host the calling thread's own count reads, in the layout
|
||
/// known, and a read of the counter sees the thread leave its core:
|
||
/// the path every reading takes. Two reads are in order, and the
|
||
/// first is this process's count: no more than the count as it
|
||
/// stands just after it.
|
||
#[test]
|
||
fn on_this_host_a_read_sees_the_reading_thread_leave_its_core() {
|
||
let thread = this_thread_energy_nj();
|
||
eprintln!("process-energy: this thread's own energy count: {thread:?} nJ");
|
||
assert!(thread.is_some(), "this thread's own energy count does not read here");
|
||
let c = MacosProcessEnergyCounter::try_new_given(CounterStatus::Live).expect("Live constructs");
|
||
let first = c.read_microjoules().expect("the reading thread was not seen to leave its core");
|
||
let now = energy_nj(this_pid()).expect("this process's count") / 1_000;
|
||
let second = c.read_microjoules().expect("the reading thread was not seen to leave its core");
|
||
assert!(first <= now && now <= second, "{first} µJ, then {now} µJ as it stands, then {second} µJ");
|
||
}
|
||
|
||
/// A host on which the reading thread's own count does not read has
|
||
/// no process counter: `Unavailable`, and the probe is not run.
|
||
#[test]
|
||
fn a_host_without_the_threads_count_has_no_process_counter() {
|
||
assert_eq!(status_where(false, || panic!("probed a counter no read could be checked on")), CounterStatus::Unavailable);
|
||
for status in [CounterStatus::Live, CounterStatus::Frozen, CounterStatus::Unavailable] {
|
||
assert_eq!(status_where(true, || status), status);
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
#[cfg(target_os = "macos")]
|
||
pub use macos_process_energy::MacosProcessEnergyCounter;
|
||
|
||
/// This host's process-energy status: whether this process's own CPU energy
|
||
/// counter ([`MacosProcessEnergyCounter`]) is live, probed once per process
|
||
/// and cached, by the rule every counter here is held to — it kept advancing
|
||
/// while the probing thread burned CPU, in at least five sample intervals, at
|
||
/// least 80% of them, over at least 300 ms. Always
|
||
/// [`CounterStatus::Unavailable`] off macOS. The first call burns about 300 ms
|
||
/// of the calling thread's CPU where the counter is live (a second if not).
|
||
pub fn process_energy_status() -> CounterStatus {
|
||
#[cfg(target_os = "macos")]
|
||
{
|
||
static STATUS: std::sync::OnceLock<CounterStatus> = std::sync::OnceLock::new();
|
||
*STATUS.get_or_init(macos_process_energy::host_status)
|
||
}
|
||
#[cfg(not(target_os = "macos"))]
|
||
{
|
||
CounterStatus::Unavailable
|
||
}
|
||
}
|
||
|
||
/// The counter for a caller whose metered work runs **on the CPU, in this
|
||
/// process** — no subprocess, no GPU or Neural Engine offload: this process's
|
||
/// own CPU energy counter where it is live ([`process_energy_status`]), and
|
||
/// otherwise whatever [`detect`] selects.
|
||
///
|
||
/// Its domain is this process's threads' CPU energy and nothing else, so a
|
||
/// span over work done anywhere else reads only this process's share of it.
|
||
/// That is why [`detect`] never selects it, and why only a caller that knows
|
||
/// its work stays in-process should ask for it.
|
||
///
|
||
/// Each read of the process counter sleeps at least 50 µs before it reads —
|
||
/// about 80 µs a read on a quiet host, a few hundred on a loaded one —
|
||
/// because it is taken only once the reading thread is seen to have left its
|
||
/// core ([`MacosProcessEnergyCounter`]). A span bracketed by two reads
|
||
/// therefore costs two such sleeps and holds its closing read's own wake-up
|
||
/// (tens of microjoules): bracket a batch of short operations with one span,
|
||
/// not each operation with its own. eoc-cascade does so: it reads this
|
||
/// counter around its neural stage, and for a cache, KV or graph hit, which
|
||
/// is under a microsecond, it reads one span around a batch of the lookup the
|
||
/// first time the tier answers and charges every hit that span's reading per
|
||
/// lookup, with no read around the hit.
|
||
pub fn detect_in_process() -> Box<dyn JouleCounter> {
|
||
detect_in_process_with(process_energy_status, detect)
|
||
}
|
||
|
||
/// The selection behind [`detect_in_process`], given the process counter's
|
||
/// status and the fallback.
|
||
#[cfg_attr(not(target_os = "macos"), allow(unused_variables))]
|
||
pub(crate) fn detect_in_process_with(
|
||
status: impl FnOnce() -> CounterStatus,
|
||
fallback: impl FnOnce() -> Box<dyn JouleCounter>,
|
||
) -> Box<dyn JouleCounter> {
|
||
#[cfg(target_os = "macos")]
|
||
if let Ok(c) = MacosProcessEnergyCounter::try_new_given(status()) {
|
||
return Box::new(c);
|
||
}
|
||
fallback()
|
||
}
|
||
|
||
/// Whether a GPU energy counter that kept advancing over its probe plausibly
|
||
/// counted its board's energy. Kept apart from NVML so it runs under test in
|
||
/// every build; used by the NVML probe (`cuda`).
|
||
#[cfg_attr(not(any(test, feature = "cuda")), allow(dead_code))]
|
||
mod gpu_plausibility {
|
||
use crate::CounterStatus;
|
||
use std::time::Duration;
|
||
|
||
/// How far past its board's maximum power limit a GPU energy counter may
|
||
/// run, on average over a probe's window, as a fraction (3/2): a board can
|
||
/// draw past its limit for milliseconds, not on average over a third of a
|
||
/// second.
|
||
pub(crate) const CEILING_MARGIN: (u128, u128) = (3, 2);
|
||
|
||
/// How much faster than its board's own power draw a GPU energy counter
|
||
/// may run over a probe's window: twice the draw's mean over the samples,
|
||
/// plus [`DRAW_SLACK_MW`]. The draw is a one-second average, so a load that
|
||
/// changed during the window can leave it behind the counter, but not by a
|
||
/// factor of two at a probe.
|
||
pub(crate) const DRAW_FACTOR: u128 = 2;
|
||
|
||
/// NVML's documented accuracy of a board's power draw, ±5 W, in mW.
|
||
pub(crate) const DRAW_SLACK_MW: u128 = 5_000;
|
||
|
||
/// Which of the two bounds a GPU counter ran past.
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||
pub(crate) enum GpuBound {
|
||
/// [`CEILING_MARGIN`] of the board's maximum power limit.
|
||
PowerLimit,
|
||
/// [`DRAW_FACTOR`] times the board's mean draw plus [`DRAW_SLACK_MW`].
|
||
MeanDraw,
|
||
}
|
||
|
||
/// What the plausibility check found, with the figures it found it from.
|
||
///
|
||
/// A `bool` was the whole of this, and it threw away everything an
|
||
/// operator needs: which bound broke, by how much, and the rate it broke
|
||
/// it at. A counter 1.1 times over a bound and one 20 times over were the
|
||
/// same answer. The figures also make the judgement reproducible — the
|
||
/// margins are constants in this module, and re-grading a past finding
|
||
/// against a different margin is only possible if the rate survived the
|
||
/// check.
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||
pub(crate) enum GpuPlausibility {
|
||
/// The counter's rate sits inside every bound whose reading was
|
||
/// present.
|
||
Plausible {
|
||
/// The rate the counter implies over the window, in milliwatts.
|
||
rate_mw: u64,
|
||
},
|
||
/// The counter ran faster than the board can draw.
|
||
CounterOverRate {
|
||
/// The rate the counter implies over the window, in milliwatts.
|
||
rate_mw: u64,
|
||
/// The bound it broke, in milliwatts.
|
||
bound_mw: u64,
|
||
/// Which bound that was.
|
||
bound: GpuBound,
|
||
},
|
||
/// The board's own telemetry read past the board's own power limit, so
|
||
/// there is no yardstick left to judge the counter against. An
|
||
/// indeterminate, not a finding about the counter — see
|
||
/// [`CounterStatus::Unjudgeable`].
|
||
BoardTelemetryUnusable {
|
||
/// The offending draw reading, in milliwatts.
|
||
draw_mw: u64,
|
||
/// The limit it exceeded (the margin-adjusted cap), in milliwatts.
|
||
limit_mw: u64,
|
||
},
|
||
}
|
||
|
||
impl GpuPlausibility {
|
||
/// Whether the counter was judged to be counting its board's energy.
|
||
///
|
||
/// `false` for an indeterminate as well as for an overrun: unjudgeable
|
||
/// is not admissible, and erring toward declining is the safe
|
||
/// direction. Use the variant itself to tell the two apart.
|
||
//
|
||
// The module's own path matches on the variants rather than call this.
|
||
#[cfg_attr(not(test), allow(dead_code))]
|
||
pub(crate) fn plausible(&self) -> bool {
|
||
matches!(self, GpuPlausibility::Plausible { .. })
|
||
}
|
||
}
|
||
|
||
/// What a GPU energy counter that counted `energy_mj` over `window` was
|
||
/// judged to be counting, beside the board's power draw `draws_mw` read
|
||
/// with the samples and its maximum power limit `ceiling_mw`. It over-ran
|
||
/// if it ran past [`CEILING_MARGIN`] of the limit, or past [`DRAW_FACTOR`]
|
||
/// times the mean draw plus [`DRAW_SLACK_MW`]. A bound whose reading is
|
||
/// missing is not applied.
|
||
///
|
||
/// Liveness says the counter is republished continuously; this says what
|
||
/// it republishes is the board's energy. On the RTX 4050 Laptop GPU
|
||
/// (driver 595.84) the total-energy counter runs several times faster than
|
||
/// the board draws for as long as it is polled: at the probe's 100 ms wait
|
||
/// it ran at 58-305 W while the board drew 5-14 W against a 65 W limit.
|
||
pub(crate) fn plausible(
|
||
energy_mj: u64,
|
||
window: Duration,
|
||
draws_mw: &[u32],
|
||
ceiling_mw: Option<u32>,
|
||
) -> GpuPlausibility {
|
||
let ns = window.as_nanos().max(1);
|
||
// mJ per second is mW.
|
||
let rate_mw = u128::from(energy_mj) * 1_000_000_000 / ns;
|
||
let clamp = |v: u128| u64::try_from(v).unwrap_or(u64::MAX);
|
||
let cap =
|
||
ceiling_mw.filter(|&l| l > 0).map(|l| u128::from(l) * CEILING_MARGIN.0 / CEILING_MARGIN.1);
|
||
|
||
// A draw reading past the board's own limit is not a reading: an RTX
|
||
// 4050 Laptop GPU reports about 590 W for a second after it wakes.
|
||
// Such a reading is dropped, and only that reading — it is one broken
|
||
// sample, not a broken bracket. Voiding the whole judgement on it is
|
||
// what this code used to do, and the hardware test on jetson-hub
|
||
// refused that shape: one spike among otherwise sane 14.8 W samples
|
||
// turned a counter running persistently at 4-30x the board's draw into
|
||
// an indeterminate, which is a worse diagnosis than the overrun it
|
||
// actually is.
|
||
//
|
||
// Dropping them also keeps the mean honest, which is what the old
|
||
// short-circuit was really for. The mean summed every reading, so a
|
||
// 590 W spike inflated it and made the mean-draw bound *more*
|
||
// permissive; that only went unnoticed because any such reading failed
|
||
// the check first.
|
||
let (usable, unusable): (Vec<u128>, Vec<u128>) =
|
||
draws_mw.iter().map(|&d| u128::from(d)).partition(|&d| cap.is_none_or(|c| d <= c));
|
||
|
||
if let Some(cap) = cap {
|
||
if rate_mw > cap {
|
||
return GpuPlausibility::CounterOverRate {
|
||
rate_mw: clamp(rate_mw),
|
||
bound_mw: clamp(cap),
|
||
bound: GpuBound::PowerLimit,
|
||
};
|
||
}
|
||
// Only when nothing usable is left is there no yardstick at all.
|
||
if usable.is_empty() && !unusable.is_empty() {
|
||
return GpuPlausibility::BoardTelemetryUnusable {
|
||
draw_mw: clamp(unusable[0]),
|
||
limit_mw: clamp(cap),
|
||
};
|
||
}
|
||
}
|
||
if !usable.is_empty() {
|
||
let mean = usable.iter().sum::<u128>() / usable.len() as u128;
|
||
let bound = DRAW_FACTOR * mean + DRAW_SLACK_MW;
|
||
if rate_mw > bound {
|
||
return GpuPlausibility::CounterOverRate {
|
||
rate_mw: clamp(rate_mw),
|
||
bound_mw: clamp(bound),
|
||
bound: GpuBound::MeanDraw,
|
||
};
|
||
}
|
||
}
|
||
GpuPlausibility::Plausible { rate_mw: clamp(rate_mw) }
|
||
}
|
||
|
||
/// What the GPU probe saw of the board beside its counter: the counter's
|
||
/// first and last reading (mJ) and when, the board's power draw read with
|
||
/// every sample, and its maximum power limit.
|
||
#[derive(Debug, Default, Clone)]
|
||
pub(crate) struct GpuWatch {
|
||
first: Option<(u64, Duration)>,
|
||
last: Option<(u64, Duration)>,
|
||
draws_mw: Vec<u32>,
|
||
pub(crate) ceiling_mw: Option<u32>,
|
||
}
|
||
|
||
impl GpuWatch {
|
||
/// Fold in one sample: the counter's reading, if it read, and the
|
||
/// board's draw, if that read, at `at`.
|
||
pub(crate) fn saw(&mut self, mj: Option<u64>, draw_mw: Option<u32>, at: Duration) {
|
||
if let Some(mj) = mj {
|
||
self.first.get_or_insert((mj, at));
|
||
self.last = Some((mj, at));
|
||
}
|
||
self.draws_mw.extend(draw_mw);
|
||
}
|
||
}
|
||
|
||
/// The status a GPU probe gives: the liveness rule's `status`, except that
|
||
/// a live counter is re-judged on what it was counting over the probe. One
|
||
/// proven to run faster than its board can draw is
|
||
/// [`CounterStatus::Implausible`]; one whose board's own telemetry was
|
||
/// unusable is [`CounterStatus::Unjudgeable`], which is a different
|
||
/// diagnosis and not a claim about the counter.
|
||
pub(crate) fn status_of(status: CounterStatus, gpu: &GpuWatch) -> CounterStatus {
|
||
if status != CounterStatus::Live {
|
||
return status;
|
||
}
|
||
match (gpu.first, gpu.last) {
|
||
(Some((a, t0)), Some((b, t1))) => {
|
||
match plausible(b.saturating_sub(a), t1.saturating_sub(t0), &gpu.draws_mw, gpu.ceiling_mw) {
|
||
GpuPlausibility::Plausible { .. } => CounterStatus::Live,
|
||
GpuPlausibility::CounterOverRate { .. } => CounterStatus::Implausible,
|
||
GpuPlausibility::BoardTelemetryUnusable { .. } => CounterStatus::Unjudgeable,
|
||
}
|
||
}
|
||
// No pair of samples to compute a rate from. Not a judgement that
|
||
// the counter is sound; liveness already passed, and there is
|
||
// nothing here to contradict it.
|
||
_ => CounterStatus::Live,
|
||
}
|
||
}
|
||
|
||
#[cfg(test)]
|
||
mod tests {
|
||
use super::*;
|
||
|
||
/// The rule with the readings of the RTX 4050 Laptop GPU on
|
||
/// jetson-hub (driver 595.84, 65 W maximum limit): polled at the
|
||
/// probe's 100 ms wait its counter ran at 58-305 W while the board drew
|
||
/// 5-14 W, which is no count of the board's energy. A counter that
|
||
/// tracks its board passes, a one-second draw lagging a load step by up
|
||
/// to twice still passes, and a bound whose reading is missing is not
|
||
/// applied.
|
||
#[test]
|
||
fn a_gpu_counter_must_count_no_faster_than_its_board_draws() {
|
||
let w = Duration::from_millis(500);
|
||
// Over a 500 ms window a counter's rate in mW is twice its mJ.
|
||
let over =
|
||
|rate_mw, bound_mw, bound| GpuPlausibility::CounterOverRate { rate_mw, bound_mw, bound };
|
||
|
||
// The figures, not just a `false`: an operator cannot act on a
|
||
// bare bool, and a finding cannot be re-graded against a changed
|
||
// margin unless the rate survived the check.
|
||
assert_eq!(
|
||
plausible(59_700, w, &[12_970, 13_520], Some(65_000)),
|
||
over(119_400, 97_500, GpuBound::PowerLimit),
|
||
"119.4 W against a 13 W draw"
|
||
);
|
||
assert_eq!(
|
||
plausible(22_000, w, &[10_400; 6], Some(65_000)),
|
||
over(44_000, 25_800, GpuBound::MeanDraw),
|
||
"44 W against a 10.4 W draw"
|
||
);
|
||
assert_eq!(
|
||
plausible(60_000, w, &[], Some(65_000)),
|
||
over(120_000, 97_500, GpuBound::PowerLimit),
|
||
"120 W on a 65 W board"
|
||
);
|
||
|
||
// A draw reading of 590 W (the board's first second after waking)
|
||
// is no reading. It is dropped, and only it: the remaining 10.3 W
|
||
// reading still judges the counter, and 88 W against it is an
|
||
// overrun. This used to assert a bare `false` meaning "a 590 W
|
||
// draw reading", which is the one thing the hardware test on
|
||
// jetson-hub refused: it turned a persistent 4-30x overrun into
|
||
// "cannot judge".
|
||
assert_eq!(
|
||
plausible(44_000, w, &[590_000, 10_300], Some(65_000)),
|
||
over(88_000, 25_600, GpuBound::MeanDraw)
|
||
);
|
||
// Dropping it also keeps the mean honest, and this pins that: the
|
||
// bound is twice 10 W plus 5 W, from the usable reading alone. Had
|
||
// the 590 W reading been summed in, the mean would be 300 W and
|
||
// the bound 605 W, which would excuse this counter.
|
||
assert_eq!(
|
||
plausible(15_000, w, &[590_000, 10_000], Some(65_000)),
|
||
over(30_000, 25_000, GpuBound::MeanDraw)
|
||
);
|
||
// Only when no usable reading is left is there no yardstick at
|
||
// all, and then the verdict is an indeterminate about the board,
|
||
// not a finding about the counter.
|
||
assert_eq!(
|
||
plausible(44_000, w, &[590_000], Some(65_000)),
|
||
GpuPlausibility::BoardTelemetryUnusable { draw_mw: 590_000, limit_mw: 97_500 }
|
||
);
|
||
// A counter past the board's own limit is a finding whatever the
|
||
// telemetry says: the rate bound is checked before the yardstick.
|
||
assert_eq!(
|
||
plausible(60_000, w, &[590_000], Some(65_000)),
|
||
over(120_000, 97_500, GpuBound::PowerLimit)
|
||
);
|
||
|
||
assert_eq!(
|
||
plausible(6_500, w, &[13_000, 13_500], Some(65_000)),
|
||
GpuPlausibility::Plausible { rate_mw: 13_000 },
|
||
"13 W against a 13 W draw"
|
||
);
|
||
assert_eq!(
|
||
plausible(150_000, w, &[280_000; 5], Some(700_000)),
|
||
GpuPlausibility::Plausible { rate_mw: 300_000 },
|
||
"300 W against a 280 W draw"
|
||
);
|
||
assert!(
|
||
plausible(102_500, w, &[100_000; 5], Some(700_000)).plausible(),
|
||
"205 W is 2 x 100 W + 5 W"
|
||
);
|
||
assert_eq!(
|
||
plausible(102_501, w, &[100_000; 5], Some(700_000)),
|
||
over(205_002, 205_000, GpuBound::MeanDraw)
|
||
);
|
||
assert!(plausible(48_750, w, &[], Some(65_000)).plausible(), "97.5 W is 1.5 x 65 W");
|
||
assert_eq!(
|
||
plausible(48_751, w, &[], Some(65_000)),
|
||
over(97_502, 97_500, GpuBound::PowerLimit)
|
||
);
|
||
// Nothing to judge it by: no bound applies, and with no limit to
|
||
// exceed no reading can be unusable either.
|
||
assert_eq!(
|
||
plausible(1_000_000, w, &[], None),
|
||
GpuPlausibility::Plausible { rate_mw: 2_000_000 },
|
||
"nothing to judge by"
|
||
);
|
||
}
|
||
|
||
/// A live counter that ran past its board is `Implausible`; one that
|
||
/// tracked it stays `Live`; a status that was not `Live` stands.
|
||
#[test]
|
||
fn a_live_gpu_counter_past_its_board_is_implausible() {
|
||
let watch = |draw_mw, ceiling_mw| {
|
||
let mut g = GpuWatch { ceiling_mw, ..GpuWatch::default() };
|
||
for k in 0..6u64 {
|
||
// 12 J every 100 ms: 120 W.
|
||
g.saw(Some(1_000_000 + 12_000 * k), Some(draw_mw), Duration::from_millis(100 * k));
|
||
}
|
||
g
|
||
};
|
||
use CounterStatus as S;
|
||
assert_eq!(status_of(S::Live, &watch(13_000, Some(65_000))), S::Implausible);
|
||
assert_eq!(status_of(S::Live, &watch(115_000, Some(700_000))), S::Live);
|
||
assert_eq!(status_of(S::Live, &GpuWatch::default()), S::Live, "nothing to judge by");
|
||
for status in [S::Unavailable, S::Frozen, S::Unjudgeable] {
|
||
assert_eq!(status_of(status, &watch(13_000, Some(65_000))), status);
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
#[cfg(feature = "cuda")]
|
||
mod nvml {
|
||
use super::*;
|
||
use crate::rail_liveness;
|
||
use nvml_wrapper::Nvml;
|
||
|
||
/// Reads GPU 0's cumulative energy via NVML: GPU telemetry, not a
|
||
/// [`JouleCounter`].
|
||
///
|
||
/// Constructed only on a host whose total-energy counter the liveness
|
||
/// probe found live and plausibly counting the board's energy
|
||
/// ([`crate::nvml_status`]); a delta it reads is a measurement of GPU 0,
|
||
/// and of nothing else. It is not a [`JouleCounter`] because eoc's
|
||
/// consumers grade any advance of a `JouleCounter` `Measured` with no
|
||
/// coverage field: a GPU-only delta graded so would claim the CPU's share
|
||
/// of the work, which it does not read. [`crate::detect`] never selects it.
|
||
pub struct NvmlCounter {
|
||
_nvml: Nvml,
|
||
device_index: u32,
|
||
}
|
||
|
||
impl NvmlCounter {
|
||
/// NVML on GPU 0, on a host where its counter is live.
|
||
/// `Err(NoCounter)` when the counter is not live; NVML's own error
|
||
/// when it does not initialise.
|
||
pub fn try_detect() -> Result<Self> {
|
||
Self::try_detect_given(crate::nvml_status())
|
||
}
|
||
|
||
/// The constructor behind [`Self::try_detect`], given this host's
|
||
/// status: every status the gate does not admit declines before NVML
|
||
/// is touched.
|
||
pub(crate) fn try_detect_given(status: CounterStatus) -> Result<Self> {
|
||
if !rail_liveness::admits_vendor_counter(status) {
|
||
return Err(Error::NoCounter);
|
||
}
|
||
Self::open().ok_or(Error::NoCounter)
|
||
}
|
||
|
||
/// NVML on GPU 0, when it initialises and the counter reads.
|
||
fn open() -> Option<Self> {
|
||
// Test builds only: count the attempt on this thread, so a test can
|
||
// pin that a declined status never gets this far.
|
||
#[cfg(test)]
|
||
OPEN_ATTEMPTS.with(|n| n.set(n.get() + 1));
|
||
let nvml = Nvml::init().ok()?;
|
||
nvml.device_by_index(0).ok()?.total_energy_consumption().ok()?;
|
||
Some(Self { _nvml: nvml, device_index: 0 })
|
||
}
|
||
|
||
/// One liveness sample: the device's total-energy counter as its one
|
||
/// rail, in its raw millijoules. `None` for a failed read.
|
||
fn sample(&self) -> Option<Vec<(String, i64)>> {
|
||
let mj = self._nvml.device_by_index(self.device_index).ok()?.total_energy_consumption().ok()?;
|
||
Some(vec![(format!("nvml device {} total energy", self.device_index), i64::try_from(mj).unwrap_or(i64::MAX))])
|
||
}
|
||
}
|
||
|
||
#[cfg(test)]
|
||
thread_local! {
|
||
/// Attempts on this thread to initialise NVML for a counter.
|
||
static OPEN_ATTEMPTS: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
|
||
}
|
||
|
||
/// How long the NVML probe waits between samples. Burning this thread's
|
||
/// CPU does not load the GPU, so the probe waits; long enough that a
|
||
/// counter the driver refreshes up to every 100 ms still advances in
|
||
/// every interval. One refreshed less often fails the 80% share and is
|
||
/// declined — estimated, never over-claimed.
|
||
const NVML_PROBE_WAIT: std::time::Duration = std::time::Duration::from_millis(100);
|
||
|
||
/// The real NVML probe. Cached by [`crate::nvml_status`], its only caller.
|
||
/// With each sample it reads the board's power draw, and once its maximum
|
||
/// power limit, so that a live counter that ran faster than the board can
|
||
/// draw is [`CounterStatus::Implausible`] ([`crate::gpu_plausibility`]).
|
||
pub(crate) fn host_status() -> CounterStatus {
|
||
let seen = std::cell::RefCell::new(crate::gpu_plausibility::GpuWatch::default());
|
||
let watch = rail_liveness::probe_counter(
|
||
|| {
|
||
let c = NvmlCounter::open()?;
|
||
seen.borrow_mut().ceiling_mw = c.max_limit_mw();
|
||
Some(c)
|
||
},
|
||
|c: &NvmlCounter| {
|
||
let sample = c.sample();
|
||
let mj = sample.as_ref().and_then(|rails| rails.first()).and_then(|(_, raw)| u64::try_from(*raw).ok());
|
||
seen.borrow_mut().saw(mj, c.draw_mw(), crate::nvml_wall_now());
|
||
sample
|
||
},
|
||
|| std::thread::sleep(NVML_PROBE_WAIT),
|
||
crate::nvml_wall_now,
|
||
crate::VENDOR_COUNTER_LIVENESS_BUDGET,
|
||
);
|
||
crate::gpu_plausibility::status_of(rail_liveness::counter_status(watch.as_ref()), &seen.into_inner())
|
||
}
|
||
|
||
impl NvmlCounter {
|
||
/// GPU 0's cumulative energy, in µJ (NVML counts millijoules). The
|
||
/// GPU's energy alone: never grade a delta of it as a figure for work
|
||
/// that also ran on the CPU.
|
||
pub fn read_gpu_microjoules(&self) -> Result<u64> {
|
||
let device = self
|
||
._nvml
|
||
.device_by_index(self.device_index)
|
||
.map_err(|e| Error::Backend(e.to_string()))?;
|
||
let mj = device
|
||
.total_energy_consumption()
|
||
.map_err(|e| Error::Backend(e.to_string()))?;
|
||
// NVML reports millijoules; convert to microjoules.
|
||
Ok(mj.saturating_mul(1000))
|
||
}
|
||
|
||
/// The board's power draw, in mW, as NVML reports it.
|
||
fn draw_mw(&self) -> Option<u32> {
|
||
self._nvml.device_by_index(self.device_index).ok()?.power_usage().ok()
|
||
}
|
||
|
||
/// The board's maximum power-management limit, in mW.
|
||
fn max_limit_mw(&self) -> Option<u32> {
|
||
let limits = self._nvml.device_by_index(self.device_index).ok()?.power_management_limit_constraints().ok()?;
|
||
(limits.max_limit > 0).then_some(limits.max_limit)
|
||
}
|
||
}
|
||
|
||
#[cfg(test)]
|
||
mod tests {
|
||
use super::*;
|
||
|
||
/// Only a live status gets as far as initialising NVML: every declined
|
||
/// status is `NoCounter` without an attempt to open it, whether or not
|
||
/// NVML would open on this host. Live tries, and constructs the
|
||
/// counter only where NVML opens.
|
||
#[test]
|
||
fn only_a_live_nvml_status_constructs_the_counter() {
|
||
let attempts = || OPEN_ATTEMPTS.with(|n| n.get());
|
||
for status in [
|
||
CounterStatus::Unavailable,
|
||
CounterStatus::Frozen,
|
||
CounterStatus::Implausible,
|
||
CounterStatus::Unjudgeable,
|
||
] {
|
||
let before = attempts();
|
||
assert!(
|
||
matches!(NvmlCounter::try_detect_given(status), Err(Error::NoCounter)),
|
||
"{status:?} constructed the NVML counter"
|
||
);
|
||
assert_eq!(attempts(), before, "under {status:?} the constructor tried to open NVML");
|
||
}
|
||
let opens = NvmlCounter::open().is_some();
|
||
let before = attempts();
|
||
assert_eq!(NvmlCounter::try_detect_given(CounterStatus::Live).is_ok(), opens);
|
||
assert_eq!(attempts(), before + 1, "given Live, the constructor did not try to open NVML");
|
||
eprintln!("nvml: opens={opens} status={:?}", crate::nvml_status());
|
||
if !opens {
|
||
assert_eq!(crate::nvml_status(), CounterStatus::Unavailable, "a status for an NVML that does not open");
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
/// Wall time on a monotonic clock, from a fixed origin in this process: the
|
||
/// NVML probe's clock (it needs no thread CPU clock, and builds on every OS).
|
||
#[cfg(feature = "cuda")]
|
||
fn nvml_wall_now() -> std::time::Duration {
|
||
static ORIGIN: std::sync::OnceLock<std::time::Instant> = std::sync::OnceLock::new();
|
||
ORIGIN.get_or_init(std::time::Instant::now).elapsed()
|
||
}
|
||
|
||
#[cfg(feature = "cuda")]
|
||
pub use nvml::NvmlCounter;
|
||
|
||
/// Runtime-detect the best available counter on this host.
|
||
///
|
||
/// Preference order: RAPL on Linux (only when every package
|
||
/// counter is live — see [`rapl_status`]) → IOReport on macOS (Energy-Model counters,
|
||
/// no sudo; only when every CPU energy rail is live — see [`ioreport_status`])
|
||
/// → powermetrics on macOS → `StubCounter`. Every hardware rung is held to
|
||
/// the same sustained-advance rule; a rung whose counter is not live is
|
||
/// skipped, so callers estimate rather than seal what it read.
|
||
///
|
||
/// It never selects NVML's GPU counter ([`NvmlCounter`], feature `cuda`),
|
||
/// which reads the GPU and not the CPU's share of the work: eoc's consumers
|
||
/// grade whatever this returns `Measured`, with no field to say a figure
|
||
/// covers part of the work, so a GPU-only figure would claim the whole.
|
||
///
|
||
/// It never selects this process's own energy counter
|
||
/// ([`MacosProcessEnergyCounter`]), whose domain stops at this process's
|
||
/// CPU threads: what eoc meters may run in a subprocess or on the GPU. A
|
||
/// caller whose work is in-process asks [`detect_in_process`].
|
||
pub fn detect() -> Box<dyn JouleCounter> {
|
||
detect_with(rapl_status, ioreport_status)
|
||
}
|
||
|
||
/// The selection behind [`detect`], given where each rung's status comes from
|
||
/// (each asked only when the ladder reaches its rung, and only on a platform
|
||
/// and build that has the rung).
|
||
// Each status is asked for only where its rung exists, so on any one platform
|
||
// and build some of them go unused.
|
||
#[allow(unused_variables)]
|
||
pub(crate) fn detect_with(
|
||
rapl: impl FnOnce() -> CounterStatus,
|
||
ioreport: impl FnOnce() -> IoReportStatus,
|
||
) -> Box<dyn JouleCounter> {
|
||
#[cfg(target_os = "linux")]
|
||
if let Ok(c) = LinuxRaplCounter::try_detect_given(rapl()) {
|
||
return Box::new(c);
|
||
}
|
||
#[cfg(target_os = "macos")]
|
||
if let Ok(c) = MacosIoReportCounter::try_detect_given(ioreport()) {
|
||
return Box::new(c);
|
||
}
|
||
#[cfg(target_os = "macos")]
|
||
if let Ok(c) = MacosPowermetricsCounter::try_detect() {
|
||
return Box::new(c);
|
||
}
|
||
Box::new(StubCounter)
|
||
}
|
||
|
||
#[cfg(test)]
|
||
mod tests {
|
||
use super::*;
|
||
|
||
#[test]
|
||
fn stub_reads_zero() {
|
||
let c = StubCounter;
|
||
assert_eq!(c.read_microjoules().unwrap(), 0);
|
||
assert_eq!(c.name(), "stub");
|
||
}
|
||
|
||
#[test]
|
||
fn detect_always_returns_something() {
|
||
let c = detect();
|
||
// Should never panic. Reading may succeed (stub) or fail (live
|
||
// counter without permission).
|
||
let _ = c.read_microjoules();
|
||
assert!(!c.name().is_empty());
|
||
}
|
||
|
||
#[cfg(not(target_os = "macos"))]
|
||
#[test]
|
||
fn ioreport_is_unavailable_off_macos() {
|
||
assert_eq!(ioreport_status(), IoReportStatus::Unavailable);
|
||
}
|
||
|
||
/// RAPL is selected only when its counter is live, and only where its
|
||
/// rung exists, on Linux. A frozen, implausible or unavailable counter is
|
||
/// never selected, whatever the other rungs say. NVML's GPU counter is
|
||
/// never selected at all: it is not a `JouleCounter`, and the ladder has
|
||
/// no rung for it, whatever its status.
|
||
#[test]
|
||
fn a_vendor_counter_is_selected_only_when_live() {
|
||
use CounterStatus as S;
|
||
let declined_ioreport = || IoReportStatus::CpuRailFrozen;
|
||
for status in [S::Unavailable, S::Frozen, S::Implausible, S::Unjudgeable] {
|
||
let picked = detect_with(|| status, declined_ioreport).name();
|
||
assert_ne!(picked, "rapl", "a {status:?} RAPL counter was selected");
|
||
}
|
||
let picked = detect_with(|| S::Live, declined_ioreport).name();
|
||
assert_eq!(picked == "rapl", cfg!(target_os = "linux"), "a live RAPL status selected {picked}");
|
||
for status in [S::Unavailable, S::Live] {
|
||
assert_ne!(detect_with(|| status, declined_ioreport).name(), "nvml");
|
||
}
|
||
}
|
||
|
||
/// `detect()` never selects this process's own energy counter, under any
|
||
/// status of any rung, nor on this host, whatever that counter's status:
|
||
/// eoc's counter brackets stages that may run in a subprocess or on the
|
||
/// GPU, which it does not see.
|
||
#[test]
|
||
fn detect_never_selects_the_process_counter() {
|
||
use CounterStatus as S;
|
||
let ioreport = [
|
||
IoReportStatus::Unavailable,
|
||
IoReportStatus::CpuRailAbsent,
|
||
IoReportStatus::CpuRailFrozen,
|
||
IoReportStatus::Live,
|
||
];
|
||
for rapl in [S::Unavailable, S::Frozen, S::Implausible, S::Unjudgeable, S::Live] {
|
||
for io in ioreport {
|
||
let picked = detect_with(|| rapl, || io).name();
|
||
assert_ne!(picked, "process-energy", "detect_with({rapl:?}, {io:?})");
|
||
}
|
||
}
|
||
let picked = detect().name();
|
||
eprintln!("detect()={picked} process_energy_status()={:?}", process_energy_status());
|
||
assert_ne!(picked, "process-energy");
|
||
}
|
||
|
||
/// `detect_in_process()` picks the process counter exactly where it is
|
||
/// live (only on macOS), and otherwise falls back to what it is given.
|
||
#[test]
|
||
fn detect_in_process_picks_the_process_counter_only_when_live() {
|
||
use CounterStatus as S;
|
||
let fallback = || -> Box<dyn JouleCounter> { Box::new(StubCounter) };
|
||
let picked = detect_in_process_with(|| S::Live, fallback).name();
|
||
assert_eq!(picked == "process-energy", cfg!(target_os = "macos"), "Live picked {picked}");
|
||
for status in [S::Unavailable, S::Frozen, S::Unjudgeable] {
|
||
assert_eq!(detect_in_process_with(|| status, fallback).name(), "stub", "{status:?}");
|
||
}
|
||
}
|
||
|
||
/// Off Linux there is no RAPL rung to probe; without `cuda`, no NVML one.
|
||
#[test]
|
||
fn a_vendor_counter_with_no_rung_is_unavailable() {
|
||
#[cfg(not(target_os = "linux"))]
|
||
assert_eq!(rapl_status(), CounterStatus::Unavailable);
|
||
#[cfg(not(feature = "cuda"))]
|
||
assert_eq!(nvml_status(), CounterStatus::Unavailable);
|
||
#[cfg(not(target_os = "macos"))]
|
||
assert_eq!(process_energy_status(), CounterStatus::Unavailable);
|
||
}
|
||
|
||
/// Against this host's RAPL package counters, where there are any: the
|
||
/// status the probe cached agrees with what they do now. Live: every one
|
||
/// advances across 200 ms of CPU work, and `detect()` selects RAPL. Not
|
||
/// live: `detect()` never selects it. Logs what it saw.
|
||
#[cfg(target_os = "linux")]
|
||
#[test]
|
||
fn the_rapl_rung_agrees_with_the_counter() {
|
||
let status = rapl_status();
|
||
let host = linux_rapl::host();
|
||
let root = std::path::Path::new(linux_rapl::POWERCAP_ROOT);
|
||
let before = linux_rapl::read_packages(root, &host.packages);
|
||
let started = std::time::Instant::now();
|
||
let mut acc: u64 = 0;
|
||
while started.elapsed() < std::time::Duration::from_millis(200) {
|
||
acc = std::hint::black_box(acc.wrapping_add(1));
|
||
}
|
||
let after = linux_rapl::read_packages(root, &host.packages);
|
||
let picked = detect().name();
|
||
eprintln!("rapl: status={status:?} packages={:?} before={before:?} after={after:?} detect()={picked}", host.packages);
|
||
if status == CounterStatus::Live {
|
||
let (b, a) = (before.expect("live counters read"), after.expect("live counters read"));
|
||
for ((p, b), a) in host.packages.iter().zip(&b).zip(&a) {
|
||
assert!(a != b, "live package zone intel-rapl:{} did not move across 200 ms of CPU work: {b} -> {a}", p.index);
|
||
}
|
||
assert_eq!(picked, "rapl", "detect() picked {picked}");
|
||
} else {
|
||
assert_ne!(picked, "rapl", "detect() selected a {status:?} RAPL counter");
|
||
}
|
||
}
|
||
}
|