meter model subprocesses where they ran: the L3 model enters the run's figure #211

Merged

dcharlot merged 12 commits from feat/meter-model-subprocesses into main 2026-10-10 22:25:18 -04:00

Conversation 0 Commits 12 Files changed 18 +2673 -291

dcharlot commented 2026-10-10 18:09:22 -04:00

Owner

Copy link

Why

A repo-wide sweep of every place that meters energy found model subprocesses metered machine-wide, which is an estimate on macOS 27, or not metered at all.

The worst case was joule-code-remediate. Its L3 model run was outside every meter, so uj_per_verified_fix left out the most expensive tier.

What

• joule-code-remediate. The L3 model's window now enters the run's total and its receipt.
  • The model is made the direct child (exec behind a gate line), so its own count is readable. It is read on its own process tree, waitid(WNOWAIT) holds it as a zombie, and only then is it reaped.
  • The model's exit loses its last slice on a core, so the reading is a lower bound. It is therefore never sealed as a hardware reading of the whole: the window is sealed as an estimate, with one record per device class.
  • On Linux the machine meter is used, as it already is for cargo.
• jouledesk-github. The per-mention model subprocess is read on its own process tree, under the same rule: never whole, sealed as an estimate. A fallback answer carries no model reading.
• l3-jouleclaw run.
  • The in-process engine is metered on this process's CPU and GPU, graded by coverage.
  • The subprocess engine is read on its own process tree.
  • A fallback span is no reading of the engine.
  • A binary that cannot run returns Unavailable.
• jouleclaw fabric calibration (fabric_calibrate, mpp_energy). These used to refuse on macOS 27. They now calibrate each fabric member on this process's own counters.
  • A figure is written only from a whole span.
  • A member that is refused or skipped has its row dropped, so it is placed on its seeded figure. Rows from two energy domains are never mixed.
• Removed settings. "Self-contained" settings that could never take effect on these paths are gone: an exiting subprocess is never read whole.
• CI. The macOS energy job runs the remediate L3 oracle test and the loader-gguf process-calibration tests. Each is gated on a line it prints only after its assertions pass. l3-jouleclaw's oracle tests run locally on Apple Silicon only; its README says so.

Evidence

• Oracles. Every new test reads the child's own ri_energy_nj through an oracle that shares no code with the meter.
  • The remediate stand-in and the jouledesk child read 0.998–0.999 of the oracle.
  • Process calibration matches the oracle exactly.
  • These held under load as well, with 110 bounded busy processes running.
• Review. An independent review found 7 defects, all fixed. Each fix has a test that fails on the old behaviour.
  • The calibration table mixed rows from two energy domains, reproduced against a copy of a real calibration file.
  • CI never ran the new oracle tests.
  • GPU energy was sealed under the CPU class.
  • Settings had no effect.
  • The fallback was mis-documented.
  • A fallback answer carried a model reading.
  • A third-party product name appeared in new text.
• Suites.
  • joule-code-remediate: 17 passed.
  • jouledesk-github: 17 passed.
  • l3-jouleclaw: all passed.
  • jouleclaw-loader-gguf: 297 with default features, 310 with rayon fabric-meter.
  • The examples build.
  • Linux cross-checks are clean.

Not covered

• Unverified in CI context: whether the macOS runner's launchd session sees the per-process GPU counter. If it does not, the calibration test takes its fallback branch and its gate fails loudly.
• fabric_calibrate ignores JOULECLAW_FABRIC_CALIBRATION, while mpp_energy honours it. This predates the branch.

🤖 Generated with Claude Code

## Why A repo-wide sweep of every place that meters energy found model subprocesses metered machine-wide, which is an estimate on macOS 27, or not metered at all. The worst case was joule-code-remediate. Its L3 model run was **outside every meter**, so `uj_per_verified_fix` left out the most expensive tier. ## What - **joule-code-remediate.** The L3 model's window now enters the run's total and its receipt. - The model is made the direct child (`exec` behind a gate line), so its own count is readable. It is read on its own process tree, `waitid(WNOWAIT)` holds it as a zombie, and only then is it reaped. - **The model's exit loses its last slice on a core, so the reading is a lower bound.** It is therefore never sealed as a hardware reading of the whole: the window is sealed as an estimate, with one record per device class. - On Linux the machine meter is used, as it already is for cargo. - **jouledesk-github.** The per-mention model subprocess is read on its own process tree, under the same rule: never whole, sealed as an estimate. A fallback answer carries no model reading. - **l3-jouleclaw `run`.** - The in-process engine is metered on this process's CPU and GPU, graded by coverage. - The subprocess engine is read on its own process tree. - A fallback span is no reading of the engine. - A binary that cannot run returns `Unavailable`. - **jouleclaw fabric calibration** (`fabric_calibrate`, `mpp_energy`). These used to refuse on macOS 27. They now calibrate each fabric member on this process's own counters. - A figure is written only from a whole span. - A member that is refused or skipped has its row dropped, so it is placed on its seeded figure. Rows from two energy domains are never mixed. - **Removed settings.** "Self-contained" settings that could never take effect on these paths are gone: an exiting subprocess is never read whole. - **CI.** The macOS energy job runs the remediate L3 oracle test and the loader-gguf process-calibration tests. Each is gated on a line it prints only after its assertions pass. l3-jouleclaw's oracle tests run locally on Apple Silicon only; its README says so. ## Evidence - **Oracles.** Every new test reads the child's own `ri_energy_nj` through an oracle that shares no code with the meter. - The remediate stand-in and the jouledesk child read 0.998–0.999 of the oracle. - Process calibration matches the oracle exactly. - These held under load as well, with 110 bounded busy processes running. - **Review.** An independent review found 7 defects, all fixed. Each fix has a test that fails on the old behaviour. - The calibration table mixed rows from two energy domains, reproduced against a copy of a real calibration file. - CI never ran the new oracle tests. - GPU energy was sealed under the CPU class. - Settings had no effect. - The fallback was mis-documented. - A fallback answer carried a model reading. - A third-party product name appeared in new text. - **Suites.** - joule-code-remediate: 17 passed. - jouledesk-github: 17 passed. - l3-jouleclaw: all passed. - jouleclaw-loader-gguf: 297 with default features, 310 with `rayon fabric-meter`. - The examples build. - Linux cross-checks are clean. ## Not covered - **Unverified in CI context:** whether the macOS runner's launchd session sees the per-process GPU counter. If it does not, the calibration test takes its fallback branch and its gate fails loudly. - **`fabric_calibrate` ignores `JOULECLAW_FABRIC_CALIBRATION`**, while `mpp_energy` honours it. This predates the branch. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 12 commits 2026-10-10 18:09:22 -04:00

joule-code-remediate: meter the L3 model's window into the run's figure ... 497309cfde

l3_pass ran the L3 command under no meter at all, so RunReport::total_uj,
uj_per_verified_fix and both receipts omitted the model's energy: the most
expensive tier of a run was the one the figure left out.

The command now runs under sh, which waits for a gate line and then execs a
plain argv, so the model is the very process a TreeMeter is rooted at and the
shell's start-up stays outside the window. The tree meter reads it at rest,
the machine-wide span opens, the gate and prompt are written, stdout is
drained, the child is waited for with WNOWAIT and the tree span stopped on its
exited record, then it is reaped. grade_l3 is the one place the window is
graded: attributed only for a whole reading of a command the operator
declared self-contained (JOULE_FIX_L3_SELF_CONTAINED, the gateway's rule);
any partial reading - every model that exits in its window has the Exited gap
- is an estimate, the reading plus the machine-wide figure (mesh's
ProcessMeter rule), never a hardware figure of the whole; no reading (Linux,
counter not live) is the machine-wide figure as a cargo window's. JCR-1
DeviceEnergy and JouleContract EnergyClass carry no coverage, so a partial
window grades the run down, never up. RunReport::l3 says what was read and
why it is partial.

Tested against an oracle sharing no code: a stand-in model (this test binary)
reports its own ri_energy_nj from inside; the window read 0.999 of it, twice.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouledesk-github: read the model subprocess on its own process tree ... 53d503dc51

Each mention runs one llama-completion subprocess, metered only by the host
meter: machine-wide, and on macOS 27 the load-average estimator. The model's
own process was never read, although this host counts each process's CPU
energy.

The model now runs behind a gate (sh waits for a line, then execs "$0" "$@",
so arguments are argv, never re-parsed): a TreeMeter rooted at that process
reads it at rest, the gate lets it go, its stdout is drained, it is waited
for with WNOWAIT, the span is stopped on its exited record, then it is
reaped. grade_work is the one place the figure is decided: HwShunt only for a
whole reading of a model the operator declared self-contained
(JOULEDESK_LLAMA_SELF_CONTAINED=1); a partial reading - every model that
exits in its window, and llama.cpp offloading to Metal - is sealed as an
Estimator, the reading plus the host meter's figure, because efficiency/1 has
no coverage field and its figure stands for the whole; a partial reading with
no host figure is no figure and is not sealed; no reading leaves the host
meter's figure as it was. The reply carries the reading and why it is partial
on its own line, on GitHub and Slack.

Tested against an oracle sharing no code: a stand-in model (this test binary
behind a script taking the model's arguments) reports its own ri_energy_nj
from inside; the reading was 0.998-0.999 of it, declared and undeclared.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

l3-jouleclaw: meter each engine on its own process, graded by coverage ... d147e2d01b

The run binary bracketed the inference with host_counter() reads: on macOS 27
the load-average estimator, so an in-process Metal decode and a llama.cpp
subprocess were both estimated from host load, although this host reads both
engines' own processes.

The in-process engine (jouleclaw-runtime, Metal kernels) is now metered by
ProcessSpanMeter(gpu_liveness_load) with SpanUses::CpuGpu: this process's CPU
plus its GPU (an exact zero with no GPU time, the rail when it owned the GPU).
The llama engine's subprocess runs behind a gate (sh waits, then execs
"$0" "$@") and is read on its own process tree by TreeMeter, waited for with
WNOWAIT before the reap (LlamaCompletionBackend::complete_metered). grade() is
the one place the receipt's energy is decided, mesh ProcessMeter's rule: a
reading that covers the whole inference keeps its grade; a partial one (GPU
shared, or a child that exited in its window, or undeclared) is an Estimator,
the reading plus the host counter's figure or the token estimate; no reading
leaves that rest as it was. The receipt has no coverage field, so a partial
reading is never sealed HwShunt.

Tested against oracles sharing no code: this process's own ri_energy_nj read
around a CPU burn (span 0.985 of it, GPU an exact zero, Whole, HwShunt), and
a stand-in llama (this test binary) reporting its own count from inside
(reading 0.998 of it, Estimator). The example workspace builds standalone.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

l3-jouleclaw: test the in-process meter through the path main runs ... db17570650

The in-process engine's span was opened inline in infer(), and the energy
test opened its own: the test exercised the meter but not the call main
makes. Both now go through in_process(), so the oracle test covers what the
binary does (SpanUses::CpuGpu included).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw-loader-gguf: calibrate the fabric on this process's own counters ... 28383e1c2c

fabric_calibrate and mpp_energy priced members only on the IOReport SoC
counter, and refused - writing nothing - wherever its CPU rail is frozen,
which is every macOS 27.0.1 host checked: the placer's calibration file could
not be produced at all.

Where IOReport declines they now do what tune_shape_live_process does: each
member's loop is one ProcessSpanMeter span, this process's CPU energy plus
its GPU energy, no idle to subtract. marginal_energy::process_figure is the
rule, pure and in one place: a figure only from a span metered whole on this
process's own counters (not the host-meter fallback), a CPU member's GPU
share an exact zero, a GPU member's GPU time spent (its energy the rail while
this process was the GPU's sole user); anything else is refused, saying why.
Rows are written at the span's grade through calibration_row, with a method
naming the new domain (process CPU + GPU, not the SoC marginal) and, for AMX,
that its Accelerate work is counted as the kernel attributes it to the
calling threads, unchecked against a rail. fabric_calibrate refuses member by
member and writes the rest (exit 2 when any was refused, 1 when none was
written); a member whose last shape was refused gets no row. Both examples
now merge into the file instead of clobbering it. The IOReport path is
unchanged where its rail is live.

Tested: the rule's every refusal, pure; and a live CPU member against an
oracle sharing no code (this process's ri_energy_nj read at the first op's
start and the last op's end, inside the span): equal, twice. On this host the
examples wrote the NEON and AMX rows (Measured) and refused the GPU members:
the GPU was shared.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw-loader-gguf: pin the process-figure refusal of a partial span of both counters ... 6786c5463d

A span can read both of this process's counters and still be partial (work
declared to use the Neural Engine). process_figure refuses it by its
coverage, which no test drove: the CPU-only case is refused by its source
as well, so dropping the coverage check passed every test.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw-loader-gguf: fabric_calibrate replaces every row of the members it measures ... 616afe4940

fabric_calibrate merged its rows by keeping every existing row of a member
it did not write this run. A member refused this run (the GPU shared with
another process) or skipped at warm-up kept the row an earlier run took,
so the file came out mixing an old SoC-marginal row (no declared grade,
loaded Benchmarked) with this run's per-process Measured rows, and the
placer, which ranks on uj_per_gflop alone, compared the two as if they
were one kind of figure, while the run said the fabric now placed on
measured figures. A member skipped at warm-up also counted as neither
written nor refused, so the run exited 0.

marginal_energy::merge_calibration now drops every row of the members the
run owns (FABRIC_CALIBRATE_MEMBERS: cpu-neon, amx, metal-gpu) before adding
the rows it measured, so a refused or skipped member falls back to its
seeded figure; rows it does not own (local/metal-mpp, which mpp_energy
writes) stay as they were. A member skipped at warm-up is now refused
(exit 2) and loses any row an earlier shape gave it. mpp_energy already
writes its one row only when measured and nothing otherwise.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

joule-code-remediate: drop JOULE_FIX_L3_SELF_CONTAINED, which could never take effect ... 351b65e813

The L3 window was attributed only for a tree reading with no gap and the
operator's JOULE_FIX_L3_SELF_CONTAINED=1. But run_l3 reads the model at its
exit (waited for, not reaped), and the tree meter marks every member that
exited in the span (TreeGap::Exited): an exiting process's record lacks its
last time on a core. The root always exits, so no reading was ever whole and
the setting, the attributed field and the whole-reading branch were dead —
while the docs offered the setting as the way to a whole reading.

The setting and its plumbing are gone, and the docs now say plainly that a
model subprocess is never read whole: its reading is a lower bound on its CPU
energy, and the window is always an estimate, the reading plus the machine-
wide figure. A reading whose tree showed no gap (only possible if the wait
for the model's exit failed) says so (NOT_READ_AT_EXIT) rather than coming
back with nothing to explain why it is partial.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

joule-code-remediate: seal the L3 window's device classes as separate records ... fe0e5d168a

The L3 window's machine-wide figure was Measurement::total_uj(), every
device class summed, and the tally sealed the run's whole figure as one
DeviceClass::Cpu record. Where the machine meter reads a GPU (an Apple GPU
rail, NVML on Linux), that GPU energy was sealed as CPU energy, which the
sandbox schema forbids (a host must not meter accelerator work as
undifferentiated Cpu work).

L3Energy now carries the machine figure per class (machine_by_class;
machine_uj is their sum), the tally keeps the non-CPU part of its estimate
per class, and meter_records() seals one Estimator record per class: the
CPU (cargo windows, the model's own reading, the machine's CPU) and each
other class the meter read. The records sum to the run's figure, so the
receipts' figure is unchanged. An attributed run is still one ModelBased
CPU record: every window attributed means no L3 window.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouledesk-github: a model process is never read whole; a fallback carries no model reading ... c4c72c3f76

JOULEDESK_LLAMA_SELF_CONTAINED could never take effect. A reading was sealed
HwShunt only with no gap and that declaration, but run_model reads the model
at its exit, and the tree meter marks every member that exited in the span
(TreeGap::Exited: an exiting process's record lacks its last time on a
core). The model always exits, so no reading was ever whole. The setting,
the Llama struct that carried it and the whole-reading branch are gone; the
docs say plainly that the reading is a lower bound and the work is always
sealed as an estimate (the reading plus the host meter's figure). A reading
whose tree showed no gap says why it is still partial (NOT_READ_AT_EXIT).

When the model binary is missing, sh's exec fails (127), the answer falls
back to the deterministic extract, yet the work still carried the reading of
that sh, so the reply named deterministic-extract beside a "model process
... read on its own process tree" line. A fallback now carries no model
reading, as the field's docs said; a test runs a missing binary.

The lines this branch added that named a third-party engine now name the
capability (a model binary that offloads to the GPU, the binary's log).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

l3-jouleclaw: grade a fallback span as no reading; never read the llama engine whole ... a64c9ad55b

The docs said a host with no per-process counter gets the token-count
estimate off macOS, but there ProcessSpanMeter falls back to detect_meter()
(the 28.6 W x wall estimator in this build), which it marks Whole, and grade()
kept that figure. A fallback span is the machine's figure, not the engine's,
so grade() now treats it as no reading and uses the rest (host_counter, else
the token estimate): one way of reading the host whichever engine ran, and
the docs hold. That is simpler than documenting a second host figure that
only the in-process engine would get.

JOULE_CODE_L3_SELF_CONTAINED could never take effect: the llama engine is
read at its exit, which the tree meter always marks (TreeGap::Exited), so no
reading was ever whole. The setting and its plumbing are gone and the docs
say the reading is a lower bound, always graded as an estimate.

A missing llama binary had become LlmError::Upstream("... exited 127"): the
binary is exec'd by the gating sh, whose failed exec reads as the model's own
failure. The binary is now resolved first (a path as given, else PATH; a
regular executable file) and one that cannot run is Unavailable, as a failed
spawn was, with nothing spawned.

The README says the oracle tests run only locally on Apple Silicon (CI tests
this workspace on Linux, where they take the fallback), and lines this branch
added name the capability, not a third-party engine.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

ci: run the L3 and fabric-meter oracle tests on the macOS energy job ... 04d0712ba1

No job ran the oracle assertions this branch added. joule-code-rs's matrix
entry runs on linux-arm64, where joule-fix's L3 energy test has no tree
meter and takes its fallback branch; jouleclaw-loader-gguf's calibration
tests need the fabric-meter feature, which no job passed, and their oracle
needs macOS.

The macOS energy job now runs both, each step in the job's style (pipefail,
a tee'd log, a grep gate on a line the test prints only where its gated
assertion ran):
- joule-code-remediate `l3_`: 'l3 model process:' (the model's reading
  checked against the stand-in's own count). The stand-in itself stays
  #[ignore]: it is the subprocess the oracle test runs (filtered to it,
  --ignored), not a test of its own, and the oracle test is not ignored.
- jouleclaw-loader-gguf --features "rayon fabric-meter": the marginal_energy
  tests ('calibration row stamped on macOS:') and tests/process_calibration
  ('process calibration: CPU member'); this also builds the fabric-meter
  examples. joule-code-rs's target dir and Cargo.lock join the job's cache.

The l3-jouleclaw example workspace stays out of this job (a heavy separate
build on the one macOS runner); its README says its oracle tests run locally.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot force-pushed feat/meter-model-subprocesses from 04d0712ba1 to 68aa43bafd 2026-10-10 22:24:44 -04:00 Compare

dcharlot merged commit 4b2d348edf into main 2026-10-10 22:25:18 -04:00

dcharlot referenced this pull request from a commit 2026-10-10 22:25:18 -04:00

Merge pull request 'meter model subprocesses where they ran: the L3 model enters the run's figure' (#211) from feat/meter-model-subprocesses into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!211

No description provided.