0db9b8be02
`sci_from_eval_report` wrapped every eval report's energy in `JouleCost::measured`, whatever produced it. An eval run against the echo backend, a vendor API's token estimate or a stub-metered cascade came out as a measured energy term. It could not do better: `EvalReport` dropped the provenance of the costs it summed, and `SciScore` had nowhere to put one. The end-to-end test `sci_for_ai` did the same in miniature. Its comment said "pretend the meter reported 2000 J on the stub backend", and it built that figure with `JouleCost::measured`. What holds now: - `EvalReport` gains `joule_source`. It is `Measured` only when every case's cost was read from a live counter. One estimated case makes the total an estimate, and so does a run with no cases, since nothing was read. Each case is separate work that ran, so an unmeasured case is not covered by any other reading. A report serialized before the field existed reads back as `Estimated` (`#[serde(default)]`): nothing in it says it was measured. - `SciScore` gains `energy_source`, the input `JouleCost`'s provenance unchanged. It has the same serde default. The numbers do not change. - `sci_from_eval_report` builds the energy term with the report's own `joule_source`. - `sci_for_ai` uses `JouleCost::estimated` for its fixed 2000 J. No counter read that figure. It now asserts that the score says so. Tests: a measured backend gives a measured report. An estimated one gives an estimated report, and so does one that turns estimated from its third answer on. An empty run is estimated. A report and a score without provenance read back as estimated. A score carries its input's provenance, with the same SCI either way. Under `eval-bridge`, a report's score keeps the report's provenance. Falsified, each mutant restored byte-for-byte: - HEAD's `JouleCost::measured(total_microjoules)` in the bridge fails `an_eval_reports_score_keeps_the_reports_provenance` (`left: Measured`, `right: Estimated`). That test is behind the `eval-bridge` feature, which nothing in the workspace enables, so a plain `cargo test --workspace` never compiles it. It was run with `cargo test -p eoc-carbon --features eval-bridge`, 39 + 8 passing. - `SciScore` ignoring its input's provenance fails two tests, one in the default build. - the runner ignoring per-case provenance fails `the_report_is_measured_only_when_every_case_was`. - `sci_for_ai` tagging its figure measured again fails that test. Not done: `CarbonAccount` still drops provenance (`account_request` is the other public way in). And a report with zero correct answers has `joules_per_correct == u64::MAX` as a sentinel, which `sci_from_eval_report` multiplies by `correct.max(1)` into a u64::MAX µJ energy term. Co-Authored-By: Claude Opus 5.5 <[email protected]>
57 lines
2.1 KiB
Rust
57 lines
2.1 KiB
Rust
//! End-to-end: joules → CarbonAccount → SCI value.
|
|
//!
|
|
//! We exercise the joule pipeline by using `eoc_meter::detect()` which
|
|
//! always returns *some* counter on every platform, then drive the SCI
|
|
//! computation with a deterministic test figure. No counter read that
|
|
//! figure, so it is an estimate, and the score says so.
|
|
|
|
use chrono::Utc;
|
|
use eoc_carbon::intensity::{CarbonIntensity, IntensityKind, ProviderKind};
|
|
use eoc_carbon::pue::Pue;
|
|
use eoc_carbon::sci_bridge::{SciInputs, SciScore};
|
|
use eoc_core::{JouleCost, JouleSource};
|
|
|
|
#[test]
|
|
fn full_request_to_sci_uses_meter_and_carbon_pipeline() {
|
|
// Step 1 — joule meter contract. detect() must always succeed and
|
|
// every counter must respond to read_microjoules().
|
|
let counter = eoc_meter::detect();
|
|
let _ = counter.read_microjoules();
|
|
assert!(!counter.name().is_empty());
|
|
|
|
// Step 2 — a fixed 2000 J for the request. No counter read it, so it
|
|
// is an estimate: the stub reads 0, and a live counter would not read
|
|
// this figure.
|
|
let joule_cost = JouleCost::estimated(2_000_000_000);
|
|
|
|
// Step 3 — grid intensity for the request's region.
|
|
let intensity = CarbonIntensity::new(
|
|
"EU-FR",
|
|
60.0,
|
|
IntensityKind::Average,
|
|
ProviderKind::Mock,
|
|
Utc::now(),
|
|
);
|
|
|
|
// Step 4 — PUE for the datacenter.
|
|
let pue = Pue::new(1.1);
|
|
|
|
// Step 5 — one inference served, R = 1.
|
|
let sci = SciScore::compute(SciInputs {
|
|
joule_cost: &joule_cost,
|
|
intensity: &intensity,
|
|
pue: &pue,
|
|
embodied_g_co2e: 0.05,
|
|
functional_units: 1.0,
|
|
});
|
|
|
|
// 2000 J * 1.1 = 2200 J = 2200 / 3.6e6 kWh ≈ 6.111e-4 kWh.
|
|
// operational = 6.111e-4 * 60 = 0.03667 gCO2e.
|
|
// SCI = (0.03667 + 0.05) / 1 = 0.08667.
|
|
assert!((sci.energy_kwh - 6.1111e-4).abs() < 1e-6, "got {}", sci.energy_kwh);
|
|
assert!((sci.operational_g_co2e - 0.03667).abs() < 1e-4);
|
|
assert_eq!(sci.embodied_g_co2e, 0.05);
|
|
assert!((sci.sci - 0.08667).abs() < 1e-4);
|
|
// The score is no more measured than the joules it was computed from.
|
|
assert_eq!(sci.energy_source, JouleSource::Estimated);
|
|
}
|