open-standards/sandbox/sandbox-rs/crates/sandbox-exec-microvm/examples/boot_microvm.rs
Transaction Science 8a3465e18d sandbox-exec-microvm: the CPU figure is every package's, read only while every package counter is live
The S5 executor's two CPU meters broke the rule sandbox-meter's RAPL probe
follows since 7cf1f036, and each sealed what it read as `HwShunt`.

`RaplMeter` read one file, `intel-rapl:0/energy_uj`, through `at(path)` or
`intel_package0()`, the constructor `prove_microvm` uses. It never read the
zone's `name`, so nothing confirmed `intel-rapl:0` is a package, and on a
two-socket host the second socket's energy was left out of a figure sealed
as the CPU's. It had no wrap handling of its own: a wrap read as a fall,
which the executor's `a >= b` happened to turn into an estimate. And nothing
checked that the counter was live: one that held, or stepped only in a batch
refresh, was sealed as a reading of the run. `MsrRaplMeter` read
`MSR_PKG_ENERGY_STATUS` through `/dev/cpu/0/msr` alone, package 0's counter,
the same one-socket figure on a multi-socket host, also with no liveness
check.

`RaplMeter` is now sandbox-meter's own RAPL probe rather than a second copy
of it. sandbox-meter gains `RaplPackages`, its `RaplProbe` behind the
liveness gate with no ungated constructor: `host()` opens the sysfs tree only
when the cached `rapl_status()` is `Live` (and is `Unavailable` off Linux),
`live_in(root, tick)` runs the same liveness run over another tree and opens
only if it is `Live`, and `cumulative_uj()` is the probe's snapshot. So the
figure is the sum of every top-level zone whose `name` is `package-<digits>`
and nothing else, every package or none, each wrap-corrected at its own
`max_energy_range_uj`; a package with no readable range does not open, and a
fall from above a package's range breaks the count for good. `RaplMeter::host()`
replaces `intel_package0()` and `at()`, which are gone; an unopened meter
samples `None` and says why (`unavailable()`), and `prove_microvm` uses it.
To make that possible on every platform, `rapl` and `liveness` are now built
everywhere, not only on Linux and under test: both take their tree or their
sampler as an argument, and only Linux opens sysfs.

`MsrRaplMeter` now reads every package, and every die of a multi-die
package, since the register is per die there. It groups every CPU under
`/sys/devices/system/cpu` by `topology/physical_package_id` and
`topology/die_id`, reads each domain through the first of its CPUs whose
`msr` device reads both registers, scales each by its own energy unit, and
sums them. It does not open when a domain reads through none of its CPUs,
when a domain's unit is zero, or when a CPU has no readable topology: an
offline CPU's package is unknown and could be one no online CPU covers, so
the sum could not be shown to be the whole CPU's. That makes a host with
offline CPUs (`nosmt` included) estimated, which is the safe side. Each
domain's counter is 32 bits, so a fall is a wrap at 2^32 counts, its
architectural range; the reserved upper half is ignored. It opens only once
every domain's raw counter has kept advancing under the same liveness rule,
through a new public `sandbox_meter::probe::counter_liveness(open, tick)`
(with `counter_liveness_tick()`, the production CPU-burning tick), probed
once per process for the host. `MsrRaplMeter::host()` replaces `package0()`
and returns why it did not open; `boot_microvm` prints it.

Tests, on fake trees in the temp directory, on macOS here and on Linux in
CI. The MSR tree is a sysfs CPU directory beside fake `msr` devices holding
the two registers at their offsets.
- `the_rapl_meter_sums_every_package_zone_and_no_other`: two sockets with
  `psys`, a top-level `dram`, subzones and the MMIO zone, opened live; with
  every zone rising, a difference is the packages' 8,000 µJ alone; package 1
  wraps at its own 65,532,610,987 µJ range (11,487 µJ); a run through
  `MicroVmExecutor` whose VMM raises both packages and `psys` seals
  `Some(3000)`, `HwShunt`.
- `the_rapl_meter_reads_nothing_unless_every_package_is_live`: package 1
  holding beside a live `psys`, `dram`, MMIO zone and package 0 leaves the
  meter unopened, `None`, and the executor seals `ModelBased`; a `psys`-only
  tree and a package with no range never open and are never ticked.
- `a_fall_from_above_a_package_range_stops_the_rapl_meter_for_good`.
- `the_msr_meter_reads_every_package_and_die`: three domains, one read
  through cpu3 because cpu2's device is absent; 256, 512 and 16,384 counts at
  ESU 14 sum to 1,046,875 µJ with package 0's reserved bits changing; a run
  through the executor seals a joule, `HwShunt`.
- `the_msr_meter_wraps_each_domain_at_32_bits`: 512 counts across the wrap.
- `the_msr_meter_declines_what_it_cannot_cover`: a domain holding beside
  live ones; an offline CPU; no `die_id`; a domain none of whose CPUs read; a
  zero unit; no CPU; the unopened ones never ticked.
- sandbox-meter: `the_public_package_counter_opens_only_over_live_packages`
  and `counter_liveness_holds_a_callers_counter_to_the_every_rail_rule`.

Falsified, each restored from a saved copy before the next:
- `RaplMeter` reading `intel-rapl:0/energy_uj` alone again (the liveness
  gate kept) fails two: "not the two packages' rise alone, left: 3000,
  right: 8000", and "a fall from above the range gave a figure, left:
  Some(10)";
- the MSR meter keeping package 0 only fails three, "not one CPU per package
  and die" among them, and the holding-domain case opens;
- the MSR wrap as a saturating difference fails "512 counts across the wrap,
  left: 0, right: 31250";
- the MSR gate opening on a Frozen status fails the declines test ("opened
  [...] with package 1 die 1 holding");
- an offline CPU skipped instead of declining fails it too.

sandbox-exec-microvm (18) and the whole sandbox-rs workspace (308 tests)
pass on macOS (M5 Max, macOS 27.0.1), as does the sandbox-conformance binary
("All classes passed with no skips"); jcp-gateway, jcp-action-receipt,
joule-code-remediate, joule-ui-interaction and joule-ui-serve pass, and
joule-ui's `gen_vectors` regenerated into a scratch directory is
byte-identical to `conformance/v1/`. `cargo clippy --all-targets -p
sandbox-meter -p sandbox-exec-microvm` is clean for aarch64-apple-darwin and
x86_64-unknown-linux-gnu; sandbox-proof is clean on the host, and its Linux
cross-check needs a cross C compiler for zstd-sys, which this machine lacks.
Not run on a Linux host with a real RAPL tree or MSRs here.

Not done here: the executor seals a zero CPU delta between two reads of a
live meter as a measured `HwShunt` zero (`a >= b`), where sandbox-meter's
span estimates a CPU counter that held.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-30 22:50:32 -04:00

145 lines
5.6 KiB
Rust
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

//! Boot a real Firecracker microVM through the reference [`FirecrackerVmm`]
//! driver, meter the boot's CPU energy on real hardware (RAPL via MSR), and
//! seal the reading as a Sandbox JCR-1 receipt — the S5 executor's full path,
//! end to end, on a Linux KVM host.
//!
//! ```bash
//! # on a Linux KVM host (e.g. AWS *.metal), as root for MSR access:
//! sudo FC_BIN=/usr/local/bin/firecracker \
//! KERNEL=/tmp/vmlinux.bin ROOTFS=/tmp/rootfs.ext4 \
//! ./boot_microvm
//! ```
//!
//! It spawns `firecracker --api-sock <sock>`, drives `PUT /boot-source`,
//! `/drives/rootfs`, `/machine-config`, and the `InstanceStart` action through
//! our unix-socket HTTP/1.1 client, lets the guest run briefly, and reads the
//! package energy counters (every package's, and every die's) before and
//! after. A genuine `HwShunt` figure where the MSRs read and every package's
//! counter proved live; an honest `ModelBased` vCPU-seconds estimate
//! otherwise.
use sandbox_exec_microvm::firecracker::{FirecrackerConfig, FirecrackerVmm, VmState, Vmm};
use sandbox_exec_microvm::power::PowerMeter;
use sandbox_meter::{DeviceEnergy, Provenance, Receipt};
use sandbox_schema::{ContentId, DeviceClass, Tier};
use std::path::Path;
use std::process::Command;
use std::time::{Duration, Instant};
fn env(key: &str, default: &str) -> String {
std::env::var(key).unwrap_or_else(|_| default.into())
}
fn main() {
let fc_bin = env("FC_BIN", "/usr/local/bin/firecracker");
let kernel = env("KERNEL", "/tmp/vmlinux.bin");
let rootfs = env("ROOTFS", "/tmp/rootfs.ext4");
let run_secs: f64 = env("RUN_SECS", "4").parse().unwrap_or(4.0);
let sock = "/tmp/fc-demo.sock";
let _ = std::fs::remove_file(sock);
// Spawn the Firecracker process listening on the API socket.
let mut child = Command::new(&fc_bin)
.args(["--api-sock", sock])
.spawn()
.unwrap_or_else(|e| panic!("spawn {fc_bin}: {e}"));
for _ in 0..100 {
if Path::new(sock).exists() {
break;
}
std::thread::sleep(Duration::from_millis(50));
}
// The meter: real RAPL via MSR where available (root on Nitro metal),
// every package's and every die's, once each has proved live; else None
// → an honest vCPU-seconds ModelBased estimate.
#[cfg(target_os = "linux")]
let meter: Option<sandbox_exec_microvm::power::MsrRaplMeter> =
match sandbox_exec_microvm::power::MsrRaplMeter::host() {
Ok(m) => Some(m),
Err(why) => {
println!("no MSR meter: {why}");
None
}
};
#[cfg(not(target_os = "linux"))]
let meter: Option<sandbox_exec_microvm::power::NoMeter> = None;
let before = meter.as_ref().and_then(|m| m.sample_uj());
println!(
"meter: {}",
if before.is_some() { "RAPL via MSR (HwShunt)" } else { "none — ModelBased fallback" }
);
let cfg = FirecrackerConfig {
api_socket: sock.into(),
kernel_image_path: kernel,
rootfs_path: rootfs,
boot_args: "console=ttyS0 reboot=k panic=1 pci=off root=/dev/vda rw".into(),
vcpus: 2,
mem_mib: 256,
};
let t0 = Instant::now();
let outcome = FirecrackerVmm.run(&cfg);
match &outcome {
Ok(o) => println!("boot: {:?} ({} vcpu-s attributed)", o.state, o.vcpu_seconds),
Err(e) => println!("boot error: {e}"),
}
// Let the guest run so the metered window covers real execution.
std::thread::sleep(Duration::from_secs_f64(run_secs));
let after = meter.as_ref().and_then(|m| m.sample_uj());
let wall = t0.elapsed();
let _ = child.kill();
let _ = child.wait();
let _ = std::fs::remove_file(sock);
let booted = matches!(outcome.map(|o| o.state), Ok(VmState::Booted | VmState::Completed));
println!("guest ran {:.2}s; booted={booted}", wall.as_secs_f64());
// Build the energy record: HwShunt from the MSR delta, else ModelBased.
let device = match (before, after) {
(Some(a), Some(b)) => {
let uj = b.saturating_sub(a);
println!("CPU energy (RAPL, every package, MSR): {:.3} J over the window", uj as f64 / 1e6);
DeviceEnergy {
class: DeviceClass::Cpu,
measured_uj: Some(uj),
estimated_uj: uj,
provenance: Provenance::HwShunt,
}
}
_ => {
// No hardware: vCPU-seconds × a coarse per-vCPU-second figure.
let est = (cfg.vcpus as u64).saturating_mul(wall.as_secs().max(1)).saturating_mul(15_000_000);
println!("CPU energy (estimate): {:.3} J", est as f64 / 1e6);
DeviceEnergy {
class: DeviceClass::Cpu,
measured_uj: None,
estimated_uj: est,
provenance: Provenance::ModelBased,
}
}
};
// Seal it as a Sandbox S5 receipt.
let receipt = Receipt::build(
ContentId::of_bytes(b"firecracker microvm boot: vmlinux + bionic rootfs"),
Tier::S5,
vec![device],
1,
[9u8; 8],
)
.expect("receipt builds");
let sk = ed25519_dalek::SigningKey::from_bytes(&[5u8; 32]);
let envelope = receipt.seal("did:sandbox:metal-demo", &sk).expect("seal");
receipt
.verify_sealed(&envelope, &sk.verifying_key(), Some("did:sandbox:metal-demo"))
.expect("verify");
std::fs::write("/tmp/microvm-receipt.cose", &envelope).ok();
println!(
"sealed S5 receipt {} ({} bytes) — verified ✓; written to /tmp/microvm-receipt.cose",
receipt.content_id().0,
envelope.len()
);
}