grade labels and fallback windows: every figure says how it was obtained #222

Merged

dcharlot merged 11 commits from fix/grade-labels-and-windows into main 2026-10-12 09:08:53 -04:00

Conversation 0 Commits 11 Files changed 24 +728 -173

dcharlot commented 2026-10-11 22:34:57 -04:00

Owner

Copy link

Why

The metering sweeps turned up smaller grading defects. In each one, a figure or label claimed better provenance than the way the figure was obtained, or the reverse. Some also mis-measured the window they billed. None of them touches a signed vector.

What

• jouleclaw-compliance. It graded every NVML figure HwShunt by counter class, including power-draw sampling. It now grades by what was actually read.
• joulehook-meter. It parsed model_based while mesh emits ModelBased, so every mesh figure bridged through usage.energy was silently under-graded to Estimator. It now reads the labels mesh actually emits and still accepts the old spelling.
• jouledesk-github and jouleclaw-harness-econ. They told readers a ModelBased figure was "measured". They now say it was modeled.
• joule-code-remediate and sandbox-meter. The meter-source string named "IOReport" whatever probe took the reading. It now names the probe that read the attribution.
• jouleclaw-llm. A cheap-tier backend's figure was always graded ModelBased, so a hard-coded 2 J per call would have been graded as a model. A backend now declares its figure's grade, and the default is Estimator.
• jouleclaw-tokenpower-bench. It defined ModelBased as "vendor TDP + utilisation". It now defines the grades as the family does: an uncalibrated model is an estimate.
• jouleclaw-energy ProcessSpanMeter. The fallback meter's window opened before the GPU snapshot and closed after it, so it included the meter's own waits for the GPU rail. That billed an 11 µs request about 9.5 J. The fallback is now read inside the snapshots, so its window is the work's, and the CPU window stays innermost. mesh-meter's docs are updated to match (mesh already bills its own estimate for this case).
• jouleclaw-fabric. calibration_file_path() is now the one place the calibration file's path is decided, used by the placer and both writers. Before, fabric_calibrate ignored JOULECLAW_FABRIC_CALIBRATION, so a calibration written under the variable was not the file the placer loaded.

Each fix has a test that fails on the old behaviour.

Evidence

• Suites:
  • jouleclaw: 360 passed;
  • jouleclaw-energy with ioreport: 122;
  • jouleclaw-loader-gguf with rayon fabric-meter: 379;
  • joulehook: 9;
  • jouledesk-github: 17;
  • joule-code-remediate: 18;
  • sandbox-meter: 185;
  • mesh-meter and mesh-openai with server,metal: 28.
• One hardware flake, unrelated: jouleclaw-energy's real-GPU test failed once while another process was using the GPU. It passed 3 of 3 when re-run, and this branch does not touch it.
• Linux cross-checks: clean for every touched library crate.

🤖 Generated with Claude Code

## Why The metering sweeps turned up smaller grading defects. In each one, a figure or label claimed better provenance than the way the figure was obtained, or the reverse. Some also mis-measured the window they billed. None of them touches a signed vector. ## What - **jouleclaw-compliance.** It graded every NVML figure `HwShunt` by counter class, including power-draw sampling. It now grades by what was actually read. - **joulehook-meter.** It parsed `model_based` while mesh emits `ModelBased`, so every mesh figure bridged through `usage.energy` was silently under-graded to `Estimator`. It now reads the labels mesh actually emits and still accepts the old spelling. - **jouledesk-github and jouleclaw-harness-econ.** They told readers a `ModelBased` figure was "measured". They now say it was modeled. - **joule-code-remediate and sandbox-meter.** The meter-source string named "IOReport" whatever probe took the reading. It now names the probe that read the attribution. - **jouleclaw-llm.** A cheap-tier backend's figure was always graded `ModelBased`, so a hard-coded 2 J per call would have been graded as a model. A backend now declares its figure's grade, and the default is `Estimator`. - **jouleclaw-tokenpower-bench.** It defined `ModelBased` as "vendor TDP + utilisation". It now defines the grades as the family does: an uncalibrated model is an estimate. - **jouleclaw-energy `ProcessSpanMeter`.** The fallback meter's window opened before the GPU snapshot and closed after it, so it included the meter's own waits for the GPU rail. That billed an 11 µs request about 9.5 J. The fallback is now read inside the snapshots, so its window is the work's, and the CPU window stays innermost. mesh-meter's docs are updated to match (mesh already bills its own estimate for this case). - **jouleclaw-fabric.** `calibration_file_path()` is now the one place the calibration file's path is decided, used by the placer and both writers. Before, `fabric_calibrate` ignored `JOULECLAW_FABRIC_CALIBRATION`, so a calibration written under the variable was not the file the placer loaded. Each fix has a test that fails on the old behaviour. ## Evidence - **Suites:** - jouleclaw: 360 passed; - jouleclaw-energy with `ioreport`: 122; - jouleclaw-loader-gguf with `rayon fabric-meter`: 379; - joulehook: 9; - jouledesk-github: 17; - joule-code-remediate: 18; - sandbox-meter: 185; - mesh-meter and mesh-openai with `server,metal`: 28. - **One hardware flake, unrelated:** jouleclaw-energy's real-GPU test failed once while another process was using the GPU. It passed 3 of 3 when re-run, and this branch does not touch it. - **Linux cross-checks:** clean for every touched library crate. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 10 commits 2026-10-11 22:34:58 -04:00

jouleclaw-compliance: grade NVML by what was read, not by counter class ... 0f1bae8e1d

MeasurementMethod::Nvml (and NvmlRapl) mapped to HwShunt unconditionally,
but NVML yields a hardware reading only from its cumulative-energy counter
read while the liveness probe judged it live; everywhere else
jouleclaw-energy integrates the sampled power draw over wall time, a vendor
power model graded ModelBased. A passport could therefore call a power x time
figure a live hardware reading and claim its inference energy "measured".

Following the IoReport / IoReportLive split, a bare Nvml / NvmlRapl now
records no liveness and is graded ModelBased (a sum is no better than its
weakest part); NvmlLive / NvmlLiveRapl name the live counter and keep
HwShunt. MeasurementMethod::nvml_from_reading maps a reading's own grade to
the method, never better.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

joulehook-meter: read Mesh's own provenance labels when bridging usage.energy ... a3bacf032b

The bridge matched only the lower-case spellings "hw_shunt" / "model_based",
but Mesh writes usage.energy.provenance with its wire labels
(mesh_schema::Provenance::jcr1_label: "HwShunt", "ModelBased", "Estimator"),
so every Mesh HwShunt or ModelBased figure fell through to the estimator
floor and was sealed into efficiency/1 under-graded.

Both spellings are now read as the same grade (the lower-case one is
JouleClaw's energy-crate display form); any other label still falls to
Estimator. The new tests feed the bridge real Mesh output
(mesh_openai::chat_completion_response with mesh_schema's jcr1_label, as
dev-dependencies) and check each grade reaches the verified receipt
unchanged. Mesh's usage.energy carries no coverage label and efficiency/1
has no coverage field, so there is no coverage label to map. The proxy's
mock upstream now writes Mesh's spelling.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouledesk-github: say a ModelBased figure was modeled, not measured ... 2ab6b9debe

obtained() put "measured via <meter>" on both HwShunt and ModelBased figures,
so a reply posted a vendor model's figure to GitHub or Slack as a
measurement. The phrase now follows the grade: a hardware reading is
"measured via" its meter, a model's figure "modeled by" it, an estimate
"estimated by" it. The reply still names the provenance label beside it.

The test that pinned "measured via `vendor-model`, provenance `model_based`"
now pins "modeled by" and that the word "measured" does not appear; the
Slack reply gets the same check, and the host test splits its ModelBased and
HwShunt arms.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw-harness-econ: report a ModelBased figure as modeled, not measured ... d84155efea

joules_obtained() printed "measured" for ModelBased as well as HwShunt, so
the local-model examples called a vendor energy model's figure (power x
time) a measurement. It now follows the grade: "measured" for a hardware
reading only, "modeled" for a model's figure, "estimated" for an estimate.
The test that pinned ModelBased to "measured" now pins "modeled".

Co-Authored-By: Claude Opus 5.5 <[email protected]>

joule-code-remediate, sandbox-meter: name the probe an attribution was read by ... ccb382ba08

joule-code-remediate's meter source said "attributed(ModelBased over IOReport
HwShunt)" for every attributed run, whatever counter its windows were read
by. Today only the Apple Silicon attribution path can attribute, so the label
happened to fit, but it was a constant, not a record: an attribution read by
RAPL, or by several probes, would still have claimed IOReport.

sandbox-meter's Attribution now carries the window's own Measurement::source
(the probe that read it, "composite" for several, "estimator" for an
estimate), so a caller names the reading by what read it. joule-code-remediate
collects the sources of its attributed windows and names them; the meter
source string moves into EnergyTally::source, where a test pins that an
attribution read by "rapl-powercap" says so and never says IOReport.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw-llm: grade a backend's figure by what the backend declares ... c70a112877

LlmCheapTier::report_provenance graded any Some(joules) as ModelBased, so a
backend that reports a bare constant (verification-tier's StaticBackend
says 2.0 J every call) would have been graded a calibrated model's figure
had it been wired to the L3 tier. A number is not evidence of a meter.

LlmBackend gains energy_provenance(), defaulting to Estimator: a backend
that read a live counter over the call says HwShunt, one with a model
calibrated on the host says ModelBased, and everything else stays an
estimate. report_provenance (now a &self method; nothing outside the crate
called it) returns that grade for a reported figure and Estimator for the
typical-cost fallback. StaticBackend keeps the default, and its comment no
longer calls its constant "real joules"; a test pins its grade.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw-tokenpower-bench: define the grades as the family does ... f9ba141751

EnergyProvenance::ModelBased was defined as "derived from vendor TDP +
utilization counters (NVML / IOReport / ROCm / RAPL)". An uncalibrated
TDP x utilization model is an estimate (JCP SPEC 5.1 rungs 5-6: only time x
a power calibrated on the host is ModelBased), so a runner following that
definition would claim a better grade than its figure has. The same text also
filed live vendor energy counters (RAPL, NVML total energy, IOReport) under
ModelBased, and defined HwShunt as an in-line shunt only.

The three definitions now follow the family: HwShunt is a shunt or a vendor
energy counter read while live; ModelBased is the device's own sampled power
integrated over time or wall time x a host-calibrated power; Estimator is a
constant or an uncalibrated model such as TDP x utilization. The crate assigns
no grade itself (callers pass the tag, and nothing in the tree uses it or
publishes its output), so there is no code path or benchmark output to
regrade.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw-energy: a fallback span's window is the work's, not the GPU waits ... 4b0d932bfc

ProcessSpanMeter read the host meter first at a span's start and last at its
end, outside both GPU snapshots. A snapshot can wait for the GPU rail to
republish (up to GPU_IDLE_WINDOW on a quiet GPU, RAIL_EDGE_WAIT on a busy
one), so a span that fell back to the host meter bracketed those waits: its
time-based estimate billed the meter's waiting, not the work (an ~11 us serve
in mesh-openai's metal example was billed ~1.5-9.5 J).

The host meter is now read between the GPU snapshot and the CPU reading at
each end: start = GPU snapshot, host meter, CPU; end = CPU, host meter, GPU
snapshot. The CPU window stays innermost (neither the snapshots' CPU time nor
the host meter's reads are in it), and the fallback window is the work's own.
The readings go through a small SpanReads seam so a test can drive the order
on a clock of its own: with snapshots waiting 0 to 1 s, a fallback span's
estimate equals the work's duration (plus one host read) and the CPU window
equals the work exactly.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

mesh-meter: the span meter's fallback window no longer holds its GPU waits ... 9a6c28ff39

ProcessMeter's docs justified billing an estimated fallback span with this
crate's EstimatorMeter by the host meter's window taking in the span meter's
GPU-rail waits. jouleclaw-energy now reads the host meter inside the GPU
snapshots, so that window is the work's own and the reason no longer holds.
The behaviour is unchanged (an estimated request is still billed by this
crate's estimator over the work's window, as the rest of a partial span is);
the comments now say so instead of describing the old window.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw-fabric: one calibration-file path for the placer and its writers ... 18124713ac

Fabric::apply_calibration_file and mpp_energy honoured
JOULECLAW_FABRIC_CALIBRATION, but fabric_calibrate always wrote
~/.jouleclaw/fabric-calibration.json, so a calibration written under the
variable was not the file the placer loaded. calibration_file_path() is
now the one place the path is decided, and the reader and both writers
use it; the rule is tested without touching the process environment.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-10-11 22:59:51 -04:00

wai-vector: the default stroke set and the test repertoire, as WSF1 sources expanded by Annex A, and the text vectors (D7-5) #224

dcharlot added 1 commit 2026-10-12 09:08:32 -04:00

jouleclaw-energy: the process-counter test asserts against an oracle, not a joule floor ... 2e13655f29

The test required 200 ms of one thread's CPU to read at least 0.1 J. What
200 ms of CPU costs depends on the core class that ran it: on an
efficiency core at a load average of 114-186 it read 0.094-0.099 J, the
counter agreeing with an independent oracle each time, and the test
failed on the host's load, not on the meter.

The burn is now held to an oracle's window around it (no more than it,
and at least three quarters of it — the counter's own reads may each miss
a scheduling quantum of the reading thread), the sleep to under a tenth
of the burn, and the parent's window while only a child works to under a
tenth of what the child burned. A counter that reads wall time at a fixed
power still fails the sleep check. 7 of 7 runs passed at load ~186,
four of them on burns the old floor would have failed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot merged commit 3fbcba6238 into main 2026-10-12 09:08:53 -04:00

dcharlot referenced this pull request from a commit 2026-10-12 09:08:55 -04:00

Merge pull request 'grade labels and fallback windows: every figure says how it was obtained' (#222) from fix/grade-labels-and-windows into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!222

No description provided.