The S5 executor's two CPU meters broke the rule sandbox-meter's RAPL probe
follows since 7cf1f036, and each sealed what it read as `HwShunt`.
`RaplMeter` read one file, `intel-rapl:0/energy_uj`, through `at(path)` or
`intel_package0()`, the constructor `prove_microvm` uses. It never read the
zone's `name`, so nothing confirmed `intel-rapl:0` is a package, and on a
two-socket host the second socket's energy was left out of a figure sealed
as the CPU's. It had no wrap handling of its own: a wrap read as a fall,
which the executor's `a >= b` happened to turn into an estimate. And nothing
checked that the counter was live: one that held, or stepped only in a batch
refresh, was sealed as a reading of the run. `MsrRaplMeter` read
`MSR_PKG_ENERGY_STATUS` through `/dev/cpu/0/msr` alone, package 0's counter,
the same one-socket figure on a multi-socket host, also with no liveness
check.
`RaplMeter` is now sandbox-meter's own RAPL probe rather than a second copy
of it. sandbox-meter gains `RaplPackages`, its `RaplProbe` behind the
liveness gate with no ungated constructor: `host()` opens the sysfs tree only
when the cached `rapl_status()` is `Live` (and is `Unavailable` off Linux),
`live_in(root, tick)` runs the same liveness run over another tree and opens
only if it is `Live`, and `cumulative_uj()` is the probe's snapshot. So the
figure is the sum of every top-level zone whose `name` is `package-<digits>`
and nothing else, every package or none, each wrap-corrected at its own
`max_energy_range_uj`; a package with no readable range does not open, and a
fall from above a package's range breaks the count for good. `RaplMeter::host()`
replaces `intel_package0()` and `at()`, which are gone; an unopened meter
samples `None` and says why (`unavailable()`), and `prove_microvm` uses it.
To make that possible on every platform, `rapl` and `liveness` are now built
everywhere, not only on Linux and under test: both take their tree or their
sampler as an argument, and only Linux opens sysfs.
`MsrRaplMeter` now reads every package, and every die of a multi-die
package, since the register is per die there. It groups every CPU under
`/sys/devices/system/cpu` by `topology/physical_package_id` and
`topology/die_id`, reads each domain through the first of its CPUs whose
`msr` device reads both registers, scales each by its own energy unit, and
sums them. It does not open when a domain reads through none of its CPUs,
when a domain's unit is zero, or when a CPU has no readable topology: an
offline CPU's package is unknown and could be one no online CPU covers, so
the sum could not be shown to be the whole CPU's. That makes a host with
offline CPUs (`nosmt` included) estimated, which is the safe side. Each
domain's counter is 32 bits, so a fall is a wrap at 2^32 counts, its
architectural range; the reserved upper half is ignored. It opens only once
every domain's raw counter has kept advancing under the same liveness rule,
through a new public `sandbox_meter::probe::counter_liveness(open, tick)`
(with `counter_liveness_tick()`, the production CPU-burning tick), probed
once per process for the host. `MsrRaplMeter::host()` replaces `package0()`
and returns why it did not open; `boot_microvm` prints it.
Tests, on fake trees in the temp directory, on macOS here and on Linux in
CI. The MSR tree is a sysfs CPU directory beside fake `msr` devices holding
the two registers at their offsets.
- `the_rapl_meter_sums_every_package_zone_and_no_other`: two sockets with
`psys`, a top-level `dram`, subzones and the MMIO zone, opened live; with
every zone rising, a difference is the packages' 8,000 µJ alone; package 1
wraps at its own 65,532,610,987 µJ range (11,487 µJ); a run through
`MicroVmExecutor` whose VMM raises both packages and `psys` seals
`Some(3000)`, `HwShunt`.
- `the_rapl_meter_reads_nothing_unless_every_package_is_live`: package 1
holding beside a live `psys`, `dram`, MMIO zone and package 0 leaves the
meter unopened, `None`, and the executor seals `ModelBased`; a `psys`-only
tree and a package with no range never open and are never ticked.
- `a_fall_from_above_a_package_range_stops_the_rapl_meter_for_good`.
- `the_msr_meter_reads_every_package_and_die`: three domains, one read
through cpu3 because cpu2's device is absent; 256, 512 and 16,384 counts at
ESU 14 sum to 1,046,875 µJ with package 0's reserved bits changing; a run
through the executor seals a joule, `HwShunt`.
- `the_msr_meter_wraps_each_domain_at_32_bits`: 512 counts across the wrap.
- `the_msr_meter_declines_what_it_cannot_cover`: a domain holding beside
live ones; an offline CPU; no `die_id`; a domain none of whose CPUs read; a
zero unit; no CPU; the unopened ones never ticked.
- sandbox-meter: `the_public_package_counter_opens_only_over_live_packages`
and `counter_liveness_holds_a_callers_counter_to_the_every_rail_rule`.
Falsified, each restored from a saved copy before the next:
- `RaplMeter` reading `intel-rapl:0/energy_uj` alone again (the liveness
gate kept) fails two: "not the two packages' rise alone, left: 3000,
right: 8000", and "a fall from above the range gave a figure, left:
Some(10)";
- the MSR meter keeping package 0 only fails three, "not one CPU per package
and die" among them, and the holding-domain case opens;
- the MSR wrap as a saturating difference fails "512 counts across the wrap,
left: 0, right: 31250";
- the MSR gate opening on a Frozen status fails the declines test ("opened
[...] with package 1 die 1 holding");
- an offline CPU skipped instead of declining fails it too.
sandbox-exec-microvm (18) and the whole sandbox-rs workspace (308 tests)
pass on macOS (M5 Max, macOS 27.0.1), as does the sandbox-conformance binary
("All classes passed with no skips"); jcp-gateway, jcp-action-receipt,
joule-code-remediate, joule-ui-interaction and joule-ui-serve pass, and
joule-ui's `gen_vectors` regenerated into a scratch directory is
byte-identical to `conformance/v1/`. `cargo clippy --all-targets -p
sandbox-meter -p sandbox-exec-microvm` is clean for aarch64-apple-darwin and
x86_64-unknown-linux-gnu; sandbox-proof is clean on the host, and its Linux
cross-check needs a cross C compiler for zstd-sys, which this machine lacks.
Not run on a Linux host with a real RAPL tree or MSRs here.
Not done here: the executor seals a zero CPU delta between two reads of a
live meter as a measured `HwShunt` zero (`a >= b`), where sandbox-meter's
span estimates a CPU counter that held.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
145 lines
5.6 KiB
Rust
145 lines
5.6 KiB
Rust
//! Boot a real Firecracker microVM through the reference [`FirecrackerVmm`]
|
||
//! driver, meter the boot's CPU energy on real hardware (RAPL via MSR), and
|
||
//! seal the reading as a Sandbox JCR-1 receipt — the S5 executor's full path,
|
||
//! end to end, on a Linux KVM host.
|
||
//!
|
||
//! ```bash
|
||
//! # on a Linux KVM host (e.g. AWS *.metal), as root for MSR access:
|
||
//! sudo FC_BIN=/usr/local/bin/firecracker \
|
||
//! KERNEL=/tmp/vmlinux.bin ROOTFS=/tmp/rootfs.ext4 \
|
||
//! ./boot_microvm
|
||
//! ```
|
||
//!
|
||
//! It spawns `firecracker --api-sock <sock>`, drives `PUT /boot-source`,
|
||
//! `/drives/rootfs`, `/machine-config`, and the `InstanceStart` action through
|
||
//! our unix-socket HTTP/1.1 client, lets the guest run briefly, and reads the
|
||
//! package energy counters (every package's, and every die's) before and
|
||
//! after. A genuine `HwShunt` figure where the MSRs read and every package's
|
||
//! counter proved live; an honest `ModelBased` vCPU-seconds estimate
|
||
//! otherwise.
|
||
|
||
use sandbox_exec_microvm::firecracker::{FirecrackerConfig, FirecrackerVmm, VmState, Vmm};
|
||
use sandbox_exec_microvm::power::PowerMeter;
|
||
use sandbox_meter::{DeviceEnergy, Provenance, Receipt};
|
||
use sandbox_schema::{ContentId, DeviceClass, Tier};
|
||
use std::path::Path;
|
||
use std::process::Command;
|
||
use std::time::{Duration, Instant};
|
||
|
||
fn env(key: &str, default: &str) -> String {
|
||
std::env::var(key).unwrap_or_else(|_| default.into())
|
||
}
|
||
|
||
fn main() {
|
||
let fc_bin = env("FC_BIN", "/usr/local/bin/firecracker");
|
||
let kernel = env("KERNEL", "/tmp/vmlinux.bin");
|
||
let rootfs = env("ROOTFS", "/tmp/rootfs.ext4");
|
||
let run_secs: f64 = env("RUN_SECS", "4").parse().unwrap_or(4.0);
|
||
let sock = "/tmp/fc-demo.sock";
|
||
let _ = std::fs::remove_file(sock);
|
||
|
||
// Spawn the Firecracker process listening on the API socket.
|
||
let mut child = Command::new(&fc_bin)
|
||
.args(["--api-sock", sock])
|
||
.spawn()
|
||
.unwrap_or_else(|e| panic!("spawn {fc_bin}: {e}"));
|
||
for _ in 0..100 {
|
||
if Path::new(sock).exists() {
|
||
break;
|
||
}
|
||
std::thread::sleep(Duration::from_millis(50));
|
||
}
|
||
|
||
// The meter: real RAPL via MSR where available (root on Nitro metal),
|
||
// every package's and every die's, once each has proved live; else None
|
||
// → an honest vCPU-seconds ModelBased estimate.
|
||
#[cfg(target_os = "linux")]
|
||
let meter: Option<sandbox_exec_microvm::power::MsrRaplMeter> =
|
||
match sandbox_exec_microvm::power::MsrRaplMeter::host() {
|
||
Ok(m) => Some(m),
|
||
Err(why) => {
|
||
println!("no MSR meter: {why}");
|
||
None
|
||
}
|
||
};
|
||
#[cfg(not(target_os = "linux"))]
|
||
let meter: Option<sandbox_exec_microvm::power::NoMeter> = None;
|
||
|
||
let before = meter.as_ref().and_then(|m| m.sample_uj());
|
||
println!(
|
||
"meter: {}",
|
||
if before.is_some() { "RAPL via MSR (HwShunt)" } else { "none — ModelBased fallback" }
|
||
);
|
||
|
||
let cfg = FirecrackerConfig {
|
||
api_socket: sock.into(),
|
||
kernel_image_path: kernel,
|
||
rootfs_path: rootfs,
|
||
boot_args: "console=ttyS0 reboot=k panic=1 pci=off root=/dev/vda rw".into(),
|
||
vcpus: 2,
|
||
mem_mib: 256,
|
||
};
|
||
|
||
let t0 = Instant::now();
|
||
let outcome = FirecrackerVmm.run(&cfg);
|
||
match &outcome {
|
||
Ok(o) => println!("boot: {:?} ({} vcpu-s attributed)", o.state, o.vcpu_seconds),
|
||
Err(e) => println!("boot error: {e}"),
|
||
}
|
||
// Let the guest run so the metered window covers real execution.
|
||
std::thread::sleep(Duration::from_secs_f64(run_secs));
|
||
let after = meter.as_ref().and_then(|m| m.sample_uj());
|
||
let wall = t0.elapsed();
|
||
let _ = child.kill();
|
||
let _ = child.wait();
|
||
let _ = std::fs::remove_file(sock);
|
||
|
||
let booted = matches!(outcome.map(|o| o.state), Ok(VmState::Booted | VmState::Completed));
|
||
println!("guest ran {:.2}s; booted={booted}", wall.as_secs_f64());
|
||
|
||
// Build the energy record: HwShunt from the MSR delta, else ModelBased.
|
||
let device = match (before, after) {
|
||
(Some(a), Some(b)) => {
|
||
let uj = b.saturating_sub(a);
|
||
println!("CPU energy (RAPL, every package, MSR): {:.3} J over the window", uj as f64 / 1e6);
|
||
DeviceEnergy {
|
||
class: DeviceClass::Cpu,
|
||
measured_uj: Some(uj),
|
||
estimated_uj: uj,
|
||
provenance: Provenance::HwShunt,
|
||
}
|
||
}
|
||
_ => {
|
||
// No hardware: vCPU-seconds × a coarse per-vCPU-second figure.
|
||
let est = (cfg.vcpus as u64).saturating_mul(wall.as_secs().max(1)).saturating_mul(15_000_000);
|
||
println!("CPU energy (estimate): {:.3} J", est as f64 / 1e6);
|
||
DeviceEnergy {
|
||
class: DeviceClass::Cpu,
|
||
measured_uj: None,
|
||
estimated_uj: est,
|
||
provenance: Provenance::ModelBased,
|
||
}
|
||
}
|
||
};
|
||
|
||
// Seal it as a Sandbox S5 receipt.
|
||
let receipt = Receipt::build(
|
||
ContentId::of_bytes(b"firecracker microvm boot: vmlinux + bionic rootfs"),
|
||
Tier::S5,
|
||
vec![device],
|
||
1,
|
||
[9u8; 8],
|
||
)
|
||
.expect("receipt builds");
|
||
let sk = ed25519_dalek::SigningKey::from_bytes(&[5u8; 32]);
|
||
let envelope = receipt.seal("did:sandbox:metal-demo", &sk).expect("seal");
|
||
receipt
|
||
.verify_sealed(&envelope, &sk.verifying_key(), Some("did:sandbox:metal-demo"))
|
||
.expect("verify");
|
||
std::fs::write("/tmp/microvm-receipt.cose", &envelope).ok();
|
||
println!(
|
||
"sealed S5 receipt {} ({} bytes) — verified ✓; written to /tmp/microvm-receipt.cose",
|
||
receipt.content_id().0,
|
||
envelope.len()
|
||
);
|
||
}
|