pool: a standard for energy-accounted multi-tenant scheduling #323

Merged

dcharlot merged 9 commits from feat/pool-energy-accounted-scheduling into main 2026-10-19 11:12:37 -04:00

Conversation 0 Commits 9 Files changed 17 +4967

dcharlot commented 2026-10-18 13:44:29 -04:00

Owner

Copy link

The first slice of an energy-native scheduling layer: a new standard, pool/1.

Why this exists

A cluster scheduler decides who gets contended capacity, and denominates that entitlement in devices — whole accelerators, fractions of one, or a share of wall clock. Devices are a proxy. The quantity actually consumed, paid for, and reported under an energy-disclosure regime is energy, and a device-count quota cannot express "this team may spend 40 MJ this window". The budget that matters ends up enforced nowhere and reconciled after the fact, if at all.

I surveyed our own tree before proposing anything: over-quota appears in zero files, fair-share in zero, gang scheduling in one (incidental), and the 65 evict hits are all cache eviction. EOC's two schedulers are work-item routing — work-stealing with joule-weighted placement, and carbon-region shifting — not multi-tenant entitlement. So the gap was on both sides.

What pool/1 is

An entitlement guarantees microjoules and devices together, and both bind. Three rules carry it, and each needs the energy unit to state at all:

1. A guess may not spend a neighbour's capacity (§4.4). A claim whose estimate is Estimator — nobody measured or modelled it — is admitted only inside its own guarantee, never against surplus borrowed from a sibling. A tenant may bear its own mis-estimate; a sibling's unspent joules are not the tenant's to gamble, and an unobserved estimate is the figure most likely to be wrong in the direction that overcommits.
2. A gang is all-or-nothing in joules, not only in slots (§2.4). Gang scheduling exists because half a distributed job is worse than none of it. The same holds for its energy: admitting part of a gang against a budget that cannot finish the whole gang spends real joules and delivers nothing.
3. Preemption follows evidence, then declaration (§5.2). Workloads with a measured cost per completed unit are reclaimed worst-first. Workloads that have completed nothing are unmeasured, not wasteful, and fall back to declared priority.

Rule 3 is why the energy unit is load-bearing rather than decorative: it ranks by cost per completed unit, which needs an outcome anchor. A scheduler that knows only device-seconds cannot express that ordering at all.

Two family disciplines, designed in rather than retrofitted

• Unknown is never an extreme. uj_per_completed_unit returns None when nothing has completed (§5.1) — not zero, which would make a freshly started workload unreclaimable, and not infinity, which would make every new workload the first victim and destroy exactly the work that would have produced the measurement.
• Pending and refused are different verdicts (§4.5). A claim that cannot fit today queues; one that can never fit as written is denied, with the reason. Reporting the second as the first is how a queue fills with work that will never run.

Admission order is normative (§4) because a different order yields a different verdict: impossibility first, then devices, then the guarantee, then borrowing. Surplus is shared by weight among every borrower under one parent — so a tenant cannot take it all by claiming first — and never crosses roots.

Scope, stated as a limit

pool/1 decides admission and preemption order. It does not place, meter or execute (§1.1), and a verdict is only as sound as the figure handed to it — which is why the grade travels into the verdict rather than sitting beside it. §7 says what it cannot do: reservation bounds what may be started, not what is ultimately spent, and nothing here detects a meter that lies about its own provenance.

Tests and falsification

10 tests, including the discriminating pair for rule 1: the same claim, the same surplus available, differing only in whether anyone measured the estimate — one admitted borrowing 5 MJ, one queued.

mutation	result
P1 rule 1 removed — a guess may borrow	only a_guess_may_not_spend_a_neighbours_surplus FAILED
P2 undefined realised cost ranked as the worst	only preemption_follows_evidence_then_declaration FAILED
P3 gang tested per member, not whole	2 FAILED
P4 the impossible queued instead of refused	only a_gang_is_tested_whole_not_per_member FAILED
P5 surplus ignores weight	only surplus_is_shared_by_weight FAILED

Each rule is independently pinned. P2 is the one worth noting: it is the family-4 error ("unknown conflated with a value") applied to a brand-new mechanism, and the test catches it.

Verification

• cargo check --workspace --all-targets (pool-rs) — exit 0
• cargo test --workspace — exit 0, 10 passed
• cargo metadata --locked — exit 0
• check-lockfiles — 39 lockfiles, all current (was 38)
• check-spec-refs — 16 standards, 1613 citations, all resolve
• check-comparative-prose — clean, 2332 files; check-no-vanity-metrics — clean, 281 files
• check-tests-in-ci — clean; the pool-rs matrix row is added to standards-build.yml on the linux runner, so the workspace is actually built rather than merely present

What this slice does not include

Placement, topology-awareness, fractional-device enforcement, admission-time workload capture, a multi-cluster control plane, and reclamation of idle workloads. Those are the next slices. This one is the entitlement model, because everything else hangs off it and it is the piece that has no analogue anywhere in joule terms.

Second slice: usage survives the process (SPEC §6)

A scheduler that restarts with empty usage hands every tenant its whole guarantee back and admits the next claim against a budget already spent. That exact defect has been found and fixed twice elsewhere in this family this week — the JCP gateway refilled every grant on restart (#305), and the JouleHook gate's budget died with each per-action process so "over budget is deny" could never fire (#310). pool/1 carries the replay from the start rather than acquiring it after someone notices.

• Usage is derived, not stored (§6.1). The ledger is the durable state; there is no second store to drift from it.
• Two events move it (§6.2). Admitted reserves the estimate; Released lifts the hold and charges the actual figure in its place, not in addition to it — the hold bounded what could start, and what the tenant spent is what it spent. A Queue or a Deny reserves nothing and is not recorded; that would make this a decision log, a different artifact with a different retention story.
• An overrun is charged at what it cost (§6.3) — which is why the grade travels with every figure: the overrun is attributable.
• The chain is verified before anything is derived (§6.4), with the index of the first broken link reported rather than a shorter history. A release naming a claim never admitted is refused at its index too: not a shorter history, a history that did not happen.

What §6.5 refuses to claim

The chain is tamper-evident as a sequence. It is not authenticated — nothing in v0.1 signs an entry, so a party able to rewrite the whole stream can rewrite a consistent one. The signed pool/1 receipt (JCR-1 canonical CBOR under COSE_Sign1, as the rest of the family seals) is a later slice, and until it lands §6.5 forbids an implementation from calling a ledger entry sealed, signed or non-repudiable. The type is named Ledger, not Receipt, for the same reason: the word would claim the half that is missing.

Falsification, second slice

mutation	result
L1 replay derives nothing	a_restart_does_not_refill_a_spent_guarantee + 1 FAILED
L2 the chain is not verified	the broken-chain and accumulated-stream tests FAILED
L3 the actual is added to the hold instead of replacing it	the release and overrun tests FAILED
L4 an unmatched release is skipped	only a_release_of_a_claim_never_admitted_is_refused FAILED
L5 resume adopts the stream but not its head	only a_resumed_ledger_continues_the_chain FAILED

Headline test is the discriminating pair this family now uses for the defect: the same tenant with its guarantee spent — replayed, still queued; not replayed, admitted.

Conformance gains four duties (§7.10–13); the honest limit moves to §8 and the two citations pointing at the old number are updated, which check-spec-refs would have caught anyway.

Totals now: 17 tests, check-spec-refs 1614 citations across 16 standards, 40 lockfiles current, cargo metadata --locked exit 0, comparative-prose and vanity-metrics clean.

Third slice: entries are sealed pool/1 JCR-1 receipts (SPEC §6.5–6.6)

§6.5 previously said what the ledger was not — tamper-evident as a sequence, unauthenticated, with the spec forbidding the word "sealed". That slice lands here and §6.5 now says what an entry is: canonical det-CBOR (RFC 8949 §4.2.1) inside a COSE_Sign1 envelope (RFC 9052, EdDSA/Ed25519, alg = -8), the steward as kid — the wire format the rest of this family seals with, so a pool/1 receipt verifies under a tool that has never heard of Pool.

The JCR-1 primitives are vendored, which is what mesh-receipt documents doing for the same reason. Vendoring also kept the dependency honest: mesh-receipt reaches into jcp-schema for the decoder, and taking that dep here would have contradicted the "self-contained" rationale in the header I had just written — so the decoder is written against JCR-1's own four-type grammar instead.

Authenticity and sequence are checked separately, because they fail separately. a_removed_entry_breaks_the_chain_though_each_entry_is_still_signed deletes an entry, asserts the survivors still verify individually, and asserts the stream is refused anyway. Signatures without a chain are a bag of receipts, not a ledger.

§6.6, canonicity is checked rather than assumed. A signature covers exact bytes, so accepting a non-canonical encoding of the same document would let two byte strings share one signature. The decoder re-encodes what it decoded and refuses any difference.

Falsification, and what it caught in my own tests

mutation	result
S2 chain not checked, signatures only	2 FAILED, incl. the removed-entry test
S3 canonicity not checked	the non-shortest-integer and key-order tests FAILED
S4 trailing-bytes guard deleted	passed at first — see below
S5 duplicate-key guard deleted	passed at first — see below

S4 and S5 initially changed no outcome. Both inputs are also caught by the §6.6 canonicity check, so is_err() passed with the guard gone and pinned nothing — my own tests asserted "refused somehow" rather than "refused for this reason". Each now matches the specific MalformedCore reason, and re-falsification shows both mutations failing their test. The guards stay rather than being deleted as redundant: they give a precise reason where the canonicity check gives a general one, and a verifier's diagnostics are part of what makes a refusal actionable.

The limit this does not remove

§8 gains it explicitly: the steward signs its own decisions. A ledger is self-attested by default — it proves this steward wrote this sequence, not that the decisions were right or that a second party saw them. The family already holds the shape of the answer elsewhere (a meter attestation signed by a key distinct from the receipt signer); pool/1 has no analogue for the decision dimension yet. Stated in the spec rather than left for a reader to find.

Totals: 31 tests, conformance §7.1–14, check-spec-refs 1614 citations / 16 standards, 40 lockfiles current, cargo metadata --locked 0, comparative-prose and vanity-metrics clean.

Fourth slice: a witness distinct from the steward (SPEC §6.7–6.8)

§8 named the limit the third slice left behind: the steward signs its own decisions. A ledger proves this steward wrote this sequence — not that the decisions were right, and not that anyone else saw them. A party able to rewrite the whole stream can rewrite a consistent one, because it holds the only key involved.

A witness is a party holding a key the steward does not. It signs the head at a stated sequence, producing a pool-witness/1 receipt: a statement that the chain reached exactly that head at exactly that length. A steward that later rewrites its history produces a different head, and the receipt stops covering it.

• Demotion, not refusal (§6.8). An unwitnessed ledger is still a ledger, graded SelfAttested rather than rejected — the shape this family already uses where a figure resolves to no independent attestation. What an implementation may not do is report the two as the same thing.
• One thing is refused outright. A witness key equal to the steward's is not a witness. verify returns WitnessIsTheSteward rather than reporting a witnessed ledger: a countersignature from the party being countersigned establishes nothing, and would be the most misleading possible reading of the word.
• Unverifiable and covers-a-different-ledger are distinct errors. The second is the interesting one — it is what a rewritten history looks like from the verifier's side — and it carries both the head the witness signed and the head actually offered.

Falsification, fourth slice

mutation	result
W1 the steward may witness itself	only the_steward_cannot_witness_itself FAILED
W2 head/seq not compared — a rewrite stays covered	only a_rewritten_history_is_no_longer_covered FAILED
W3 the steward field not checked	only a_receipt_for_another_steward_is_refused FAILED
W4 self-attested reported as witnessed	only self_attested_is_a_grade_not_an_error FAILED

Each rule is pinned by exactly one test, and the headline test is the one the slice exists for: a steward rewrites its history, every surviving entry still verifies individually, and the witness's receipt no longer covers the head.

Fifth slice: attributing a shared device's energy (SPEC §2.5)

Serving several workloads from one physical accelerator is how a scheduler stops stranding capacity, and the memory fence that bounds each tenant is enforceable because memory is partitioned. Energy is not. A device counter reports one figure for the whole device and has no notion of which tenant caused which joule — so a per-tenant figure is a split of a measurement, not a measurement. Two rules follow, and share exists to make them hold rather than to be remembered:

• Provenance degrades on division. A HwShunt device total becomes ModelBased the moment two or more occupants divide it, because the division is a model however good the counter was. It never improves — dividing an Estimator guess does not make it a model — and dividing by one is not a division, so a sole occupant keeps the device's own grade. An implementation may not report a shared device's per-tenant figure as HwShunt: that grade asserts a counter was read for this work, and on a shared device none was.
• The shares sum to the measurement, exactly. Integer microjoules, largest remainder, ties broken on the claim id, arithmetic in u128 so a wide device total cannot overflow the weighting. Conservation is what makes a tenant's figure reconcilable against the device that produced it at all.

Attribution refuses a device with no occupants, or one whose occupants all carry zero weight: there is then no basis to divide on, and discarding the figure would lose energy the counter really read. The weight basis stays the operator's to choose (allotted memory, time slices, partitions) — pool/1 requires only that one basis covers every occupant of one device and that the result is graded as the model it is.

Falsification, fifth slice — and the decoration it found

mutation	result
M1 no provenance degradation on division	provenance_degrades_on_division_but_not_on_a_sole_occupant + 2 FAILED
M2 no largest-remainder leftover pass	the_shares_sum_to_the_measurement_exactly + 2 FAILED
M3 no claim-id tie-break on equal remainders	only the_leftover_is_distributed_deterministically FAILED
M4 reconcile takes the best grade instead of the worst	only reconciling_across_devices_takes_the_weakest_grade FAILED
M5 no refusal of a weightless occupant set	only a_device_with_nobody_on_it_and_a_weightless_set_are_refused FAILED

M4 is the one worth reading. reconcile folds shares back to a total and keeps the weaker grade, because a sum is no better graded than its worst term. My first test for that fed it shares that all carried the same grade — so min and max were indistinguishable and the test pinned nothing. Reconciling across devices is where the grade actually matters (one device read by a counter, another only modelled), and that case is reachable, so it now has a test of its own. Note that the original same-grade test does not appear in M4's failure list: only the new one catches it.

Verification after the fifth slice

• cargo check --workspace --all-targets --locked (pool-rs) — exit 0
• cargo test --workspace --locked — exit 0, 51 passed
• cargo clippy --workspace --all-targets --no-deps --locked — clean
• cargo metadata --locked — exit 0
• check-spec-refs — 1614 citations across 16 standards, all resolve
• check-lockfiles — 40 lockfiles, all current
• check-tests-in-ci — every test file is run by a job, and none can run empty
• check-comparative-prose — clean (2332 files); check-no-vanity-metrics — clean (281 files)

Conformance now runs §7.1–20. Still out of scope and named as such in §1.1 and §8: placement and topology-awareness, admission-time workload capture, idle reclamation, and a multi-cluster control plane.

Sixth slice: a guess earns its way out (SPEC §2.6)

Rule 1 refuses a guess the right to spend a neighbour's capacity. Reviewing the branch as a whole, that rule had no exit: a tenant nobody has measured can never be measured, because it can never run against surplus. As written it was a permanent penalty, not an incentive.

The evidence was already in the ledger — every completed claim recorded what it actually consumed, at the grade of whatever read it. What was missing was what the reservation was for, so Event::Admitted now records the claim's class: the operator's assertion that two claims are the same kind of work. It goes in the entry rather than in a scheduler-side map because §6.1 makes the ledger the only durable state, and such a map is exactly the second store that can drift from it.

observe::derive turns that history into an estimate under four rules:

• A guess is not evidence. Estimator observations are excluded, not averaged in. Counting them would launder guesses into a grade asserting they were observed — and would make the exit circular, since a tenant would earn an observed estimate merely by having run. It earns one by being measured.
• A derived estimate is a model, never a reading. Capped at ModelBased even where every observation came from a hardware counter, because a figure for the next run is a prediction. This is the same cap share applies to a divided device total, for the same reason. It is a ceiling, not a floor: modelled inputs give a modelled output.
• One run is not a sample. Below the required count nothing is derived, and the counts are reported separately — observations of the class, and how many were evidence — because history that is all guesses needs a meter while little history needs patience. required: 0 floors at one, so it can never mean "derive from nothing". The count is a policy value, and §2.6 says so rather than pretending pool/1 knows the right number for a workload it has never seen.
• The absence of a class is not a class. An empty class asserts nothing about comparability and is refused rather than pooling unlike work into one figure.

The reservation takes the largest observed figure, not the mean: under-reserving is the failure rule 1 exists to prevent, and a mean under-reserves roughly half the time by construction. The median reported beside it is the lower middle value at an even count, so no reported figure is an average nothing measured. §2.6 states the limit — the maximum of a small sample underestimates the true tail, so this is better evidence than a guess, not a bound on what the work can cost.

ledger::observations reads evidence back out of a verified stream. Only a matched Released yields an observation: an open admission is a reservation, not a result, and reserving a figure is not evidence that the work costs it. The stream's first pass is now extracted as verified_events and shared with replay — a second verification path is a second place for the checks to drift.

Falsification, sixth slice

mutation	result
O1 guesses counted as evidence	only a_guess_is_not_evidence_however_much_of_it_there_is FAILED
O2 derived estimate keeps the counter's grade	only a_derived_estimate_is_never_graded_as_a_reading FAILED
O3 no minimum evidence — one run is a sample	one_run_is_not_a_sample + 3 FAILED
O4 required: 0 means derive from nothing	only zero_required_still_needs_one_observation FAILED
O5 the reservation takes the mean	the_reservation_takes_the_largest_not_the_mean + 2 FAILED
O6 an empty class pooled rather than refused	only an_empty_class_is_refused_rather_than_pooled FAILED
O7 observations read from an unverified stream	only a_broken_stream_yields_no_evidence_at_all FAILED
O8 an open admission counts as evidence	an_open_admission_is_not_evidence + 1 FAILED

O7 is worth a note on method: my first attempt at it did not compile, so it initially reported as surviving. A mutation that does not build tests nothing, and I rewrote it until it compiled before reading the result.

The headline test runs end to end through a sealed ledger: three measured train claims leave the tenant inside its 5 MJ guarantee, the derivation produces 1 500 002 µJ from that history, and the same number of microjoules is then offered twice — queued as a guess, admitted by borrowing once it is derived from the ledger's own measured history. The figure comes out of the history, not out of the test.

Totals after the sixth slice

• cargo check --workspace --all-targets --locked — exit 0; cargo test --workspace --locked — exit 0, 63 passed
• cargo clippy --workspace --all-targets --no-deps --locked -- -D warnings — clean; cargo metadata --locked — exit 0
• check-spec-refs — 1619 citations across 16 standards, all resolve
• check-lockfiles — 40 lockfiles current; check-tests-in-ci — clean
• check-comparative-prose and check-no-vanity-metrics — clean
• Conformance now runs §7.1–24

A side note on that spec-reference count. It had not moved across the two previous slices, which turned out to be because the guard only validates citations written as SPEC §x.y in .rs files, and this crate's doc comments used a bare §. They were therefore unchecked. Rewritten into the validated form, the count moved 1614 → 1619 and all resolve — the citations are now actually verified rather than merely present.

Still out of scope and named as such in §1.1 and §8: placement and topology-awareness, idle reclamation, and a multi-cluster control plane.

Seventh slice: reclaiming idle capacity without reclaiming on a missing number (SPEC §2.7)

A workload holding accelerators while consuming almost no energy is holding capacity it is not using. A device ledger cannot see that — a held device is a held device, saturated or parked at an interactive prompt — so this is a question only an energy-denominated scheduler can ask.

It is also the most dangerous question in this standard, because the obvious implementation destroys running work. "this workload spent 0 µJ" arrives identically from a workload that is genuinely parked and from a meter that is not there:

  meter unavailable → 0 µJ → "idle" → reclaim → running work destroyed

That is this family's "unknown conflated with a value" defect in its most destructive form, so the module is built around refusing it rather than acquiring the refusal later:

• Unknown is not idle. An Estimator figure cannot establish idleness whatever its value, and the grade is tested before the value meets any threshold — so no threshold setting lets a guess through, including one high enough to catch everything. A workload is reclaimed on evidence that it is idle, never on the absence of evidence that it is busy.
• Idleness is a rate over a stated window, never an instant. Any workload is momentarily between steps. An observation shorter than the policy's window yields no verdict: startup is not idleness. An empty window yields no rate — mean_uw() returns None, not zero, because zero is the one value that reads as idle.
• Nothing held, nothing to reclaim.
• Four verdicts, and only one permits reclamation. A workload with no verdict is absent from reclamation_order, not ranked last in it: it is not a worse candidate than an idle workload, it is not a candidate.

The check order is normative: nothing held, then the window, then the grade, then the rate. A later check never rescues an earlier refusal.

Why there is no default threshold

A parked accelerator still draws power. A threshold has to sit above the hardware's idle draw or nothing is ever idle, and below its working draw or everything is — and both are properties of the device, not of this standard. pool/1 declines to guess them, so IdlePolicy::default() reclaims nothing. That is deliberate: the failure of an unconfigured destructive action should be a missed optimisation, not destroyed work.

This is also not preemption. §5.2 ranks who is spending badly under contention by cost per completed unit; this asks who is not spending at all. A workload burning energy while completing nothing is thrashing, not idle, and §5.2 is the question that catches it.

Falsification, seventh slice

mutation	result
R1 a guess may establish idleness	a_guess_of_zero_is_not_idle_though_a_reading_of_zero_is + 2 FAILED
R2 grade tested after the value meets the threshold	no_floor_setting_lets_a_guess_be_reclaimed + 2 FAILED
R3 no minimum window — an instant is a verdict	a_workload_younger_than_the_window_gets_no_verdict + 1 FAILED
R4 an empty window is a rate of zero	only an_empty_window_has_no_rate_rather_than_a_rate_of_zero FAILED
R5 the unmeasured ranked last instead of absent	the_unmeasured_are_absent_from_the_order_not_ranked_last + 1 FAILED
R6 nothing-held treated as idle	only a_workload_holding_nothing_is_not_a_candidate FAILED
R7 the threshold is inclusive at its boundary	the_floor_is_exclusive_at_its_boundary + 1 FAILED
R8 the default policy reclaims rather than declining	only the_default_policy_reclaims_nothing FAILED
R9 the rate conversion overflows in u64	only a_long_window_does_not_overflow_the_rate FAILED

R2 is the one worth reading, because it is the mutation a careless refactor would actually produce: the grade check is still present, just moved below the threshold comparison. The behaviour is then correct for every workload except the ones that matter, and no_floor_setting_lets_a_guess_be_reclaimed is the test that pins the ordering rather than merely the presence of the check.

As with O7 in the previous slice, R8's first form did not compile and so initially read as surviving. A mutation that does not build tests nothing; it was rewritten until it compiled before the result was read.

The headline test is the discriminating pair: the same workload, the same devices, the same window, the same figure of zero — reclaimable when a counter read it, protected when nobody did.

Totals, branch as it stands

• cargo check --workspace --all-targets --locked — exit 0; cargo test --workspace --locked — exit 0, 74 passed
• cargo clippy --workspace --all-targets --no-deps --locked -- -D warnings — clean
• cargo doc --no-deps — clean; this slice also fixes three broken intra-doc links left by earlier ones (WitnessReceipt, the derive function/attribute ambiguity, and a public doc linking the private first pass)
• cargo metadata --locked — exit 0
• check-spec-refs — 1621 citations across 16 standards, all resolve
• check-lockfiles — 40 lockfiles current; check-tests-in-ci — clean
• check-comparative-prose and check-no-vanity-metrics — clean
• Conformance now runs §7.1–28

Remaining out of scope, and named as such in §1.1 and §8: placement and topology-awareness, and a multi-cluster control plane.

🤖 Generated with Claude Code

The first slice of an energy-native scheduling layer: a new standard, `pool/1`. ## Why this exists A cluster scheduler decides who gets contended capacity, and denominates that entitlement in **devices** — whole accelerators, fractions of one, or a share of wall clock. Devices are a proxy. The quantity actually consumed, paid for, and reported under an energy-disclosure regime is **energy**, and a device-count quota cannot express *"this team may spend 40 MJ this window"*. The budget that matters ends up enforced nowhere and reconciled after the fact, if at all. I surveyed our own tree before proposing anything: `over-quota` appears in **zero** files, `fair-share` in **zero**, gang scheduling in one (incidental), and the 65 `evict` hits are all cache eviction. EOC's two schedulers are work-item routing — work-stealing with joule-weighted placement, and carbon-region shifting — not multi-tenant entitlement. So the gap was on both sides. ## What `pool/1` is An entitlement guarantees **microjoules and devices together**, and both bind. Three rules carry it, and each needs the energy unit to state at all: 1. **A guess may not spend a neighbour's capacity** (§4.4). A claim whose estimate is `Estimator` — nobody measured or modelled it — is admitted only inside its own guarantee, never against surplus borrowed from a sibling. A tenant may bear its own mis-estimate; a sibling's unspent joules are not the tenant's to gamble, and an unobserved estimate is the figure most likely to be wrong in the direction that overcommits. 2. **A gang is all-or-nothing in joules, not only in slots** (§2.4). Gang scheduling exists because half a distributed job is worse than none of it. The same holds for its energy: admitting part of a gang against a budget that cannot finish the whole gang spends real joules and delivers nothing. 3. **Preemption follows evidence, then declaration** (§5.2). Workloads with a measured cost per **completed** unit are reclaimed worst-first. Workloads that have completed nothing are *unmeasured*, not wasteful, and fall back to declared priority. Rule 3 is why the energy unit is load-bearing rather than decorative: it ranks by cost per *completed* unit, which needs an outcome anchor. A scheduler that knows only device-seconds cannot express that ordering at all. ## Two family disciplines, designed in rather than retrofitted - **Unknown is never an extreme.** `uj_per_completed_unit` returns `None` when nothing has completed (§5.1) — not zero, which would make a freshly started workload unreclaimable, and not infinity, which would make every new workload the first victim and destroy exactly the work that would have produced the measurement. - **Pending and refused are different verdicts** (§4.5). A claim that cannot fit *today* queues; one that can never fit *as written* is denied, with the reason. Reporting the second as the first is how a queue fills with work that will never run. Admission order is **normative** (§4) because a different order yields a different verdict: impossibility first, then devices, then the guarantee, then borrowing. Surplus is shared by weight among every borrower under one parent — so a tenant cannot take it all by claiming first — and never crosses roots. ## Scope, stated as a limit `pool/1` decides admission and preemption order. It does not place, meter or execute (§1.1), and a verdict is only as sound as the figure handed to it — which is why the grade travels **into** the verdict rather than sitting beside it. §7 says what it cannot do: reservation bounds what may be *started*, not what is ultimately *spent*, and nothing here detects a meter that lies about its own provenance. ## Tests and falsification 10 tests, including the discriminating pair for rule 1: the same claim, the same surplus available, differing only in whether anyone measured the estimate — one admitted borrowing 5 MJ, one queued. | mutation | result | |---|---| | **P1** rule 1 removed — a guess may borrow | **only** `a_guess_may_not_spend_a_neighbours_surplus` FAILED | | **P2** undefined realised cost ranked as the worst | **only** `preemption_follows_evidence_then_declaration` FAILED | | **P3** gang tested per member, not whole | **2** FAILED | | **P4** the impossible queued instead of refused | **only** `a_gang_is_tested_whole_not_per_member` FAILED | | **P5** surplus ignores weight | **only** `surplus_is_shared_by_weight` FAILED | Each rule is independently pinned. P2 is the one worth noting: it is the family-4 error ("unknown conflated with a value") applied to a brand-new mechanism, and the test catches it. ## Verification - `cargo check --workspace --all-targets` (`pool-rs`) — exit **0** - `cargo test --workspace` — exit **0**, 10 passed - `cargo metadata --locked` — exit **0** - `check-lockfiles` — 39 lockfiles, all current (was 38) - `check-spec-refs` — **16** standards, 1613 citations, all resolve - `check-comparative-prose` — clean, 2332 files; `check-no-vanity-metrics` — clean, 281 files - `check-tests-in-ci` — clean; the `pool-rs` matrix row is added to `standards-build.yml` on the `linux` runner, so the workspace is actually built rather than merely present ## What this slice does not include Placement, topology-awareness, fractional-device enforcement, admission-time workload capture, a multi-cluster control plane, and reclamation of idle workloads. Those are the next slices. This one is the entitlement model, because everything else hangs off it and it is the piece that has no analogue anywhere in joule terms. ## Second slice: usage survives the process (SPEC §6) A scheduler that restarts with empty usage hands every tenant its whole guarantee back and admits the next claim against a budget already spent. **That exact defect has been found and fixed twice elsewhere in this family this week** — the JCP gateway refilled every grant on restart (#305), and the JouleHook gate's budget died with each per-action process so "over budget is deny" could never fire (#310). `pool/1` carries the replay from the start rather than acquiring it after someone notices. - **Usage is derived, not stored** (§6.1). The ledger is the durable state; there is no second store to drift from it. - **Two events move it** (§6.2). `Admitted` reserves the estimate; `Released` lifts the hold and charges the **actual** figure *in its place*, not in addition to it — the hold bounded what could start, and what the tenant spent is what it spent. A `Queue` or a `Deny` reserves nothing and is not recorded; that would make this a decision log, a different artifact with a different retention story. - **An overrun is charged at what it cost** (§6.3) — which is why the grade travels with every figure: the overrun is attributable. - **The chain is verified before anything is derived** (§6.4), with the index of the first broken link reported rather than a shorter history. A release naming a claim never admitted is refused at its index too: not a shorter history, a history that did not happen. ### What §6.5 refuses to claim The chain is tamper-evident **as a sequence**. It is **not authenticated** — nothing in v0.1 signs an entry, so a party able to rewrite the whole stream can rewrite a consistent one. The signed `pool/1` receipt (JCR-1 canonical CBOR under COSE_Sign1, as the rest of the family seals) is a later slice, and until it lands §6.5 forbids an implementation from calling a ledger entry sealed, signed or non-repudiable. The type is named `Ledger`, not `Receipt`, for the same reason: the word would claim the half that is missing. ### Falsification, second slice | mutation | result | |---|---| | **L1** replay derives nothing | `a_restart_does_not_refill_a_spent_guarantee` + 1 FAILED | | **L2** the chain is not verified | the broken-chain and accumulated-stream tests FAILED | | **L3** the actual is added to the hold instead of replacing it | the release and overrun tests FAILED | | **L4** an unmatched release is skipped | **only** `a_release_of_a_claim_never_admitted_is_refused` FAILED | | **L5** resume adopts the stream but not its head | **only** `a_resumed_ledger_continues_the_chain` FAILED | Headline test is the discriminating pair this family now uses for the defect: the same tenant with its guarantee spent — replayed, still queued; not replayed, admitted. Conformance gains four duties (§7.10–13); the honest limit moves to §8 and the two citations pointing at the old number are updated, which `check-spec-refs` would have caught anyway. **Totals now:** 17 tests, `check-spec-refs` 1614 citations across 16 standards, 40 lockfiles current, `cargo metadata --locked` exit 0, comparative-prose and vanity-metrics clean. ## Third slice: entries are sealed `pool/1` JCR-1 receipts (SPEC §6.5–6.6) §6.5 previously said what the ledger was *not* — tamper-evident as a sequence, unauthenticated, with the spec forbidding the word "sealed". That slice lands here and §6.5 now says what an entry **is**: canonical det-CBOR (RFC 8949 §4.2.1) inside a COSE_Sign1 envelope (RFC 9052, EdDSA/Ed25519, `alg = -8`), the steward as `kid` — the wire format the rest of this family seals with, so a `pool/1` receipt verifies under a tool that has never heard of Pool. The JCR-1 primitives are **vendored**, which is what `mesh-receipt` documents doing for the same reason. Vendoring also kept the dependency honest: `mesh-receipt` reaches into `jcp-schema` for the decoder, and taking that dep here would have contradicted the "self-contained" rationale in the header I had just written — so the decoder is written against JCR-1's own four-type grammar instead. **Authenticity and sequence are checked separately, because they fail separately.** `a_removed_entry_breaks_the_chain_though_each_entry_is_still_signed` deletes an entry, asserts the survivors *still verify individually*, and asserts the stream is refused anyway. Signatures without a chain are a bag of receipts, not a ledger. **§6.6, canonicity is checked rather than assumed.** A signature covers exact bytes, so accepting a non-canonical encoding of the same document would let two byte strings share one signature. The decoder re-encodes what it decoded and refuses any difference. ### Falsification, and what it caught in my own tests | mutation | result | |---|---| | **S2** chain not checked, signatures only | 2 FAILED, incl. the removed-entry test | | **S3** canonicity not checked | the non-shortest-integer and key-order tests FAILED | | **S4** trailing-bytes guard deleted | **passed at first** — see below | | **S5** duplicate-key guard deleted | **passed at first** — see below | S4 and S5 initially changed no outcome. Both inputs are *also* caught by the §6.6 canonicity check, so `is_err()` passed with the guard gone and pinned nothing — my own tests asserted "refused somehow" rather than "refused for this reason". Each now matches the specific `MalformedCore` reason, and re-falsification shows both mutations failing their test. The guards stay rather than being deleted as redundant: they give a precise reason where the canonicity check gives a general one, and a verifier's diagnostics are part of what makes a refusal actionable. ### The limit this does not remove §8 gains it explicitly: **the steward signs its own decisions.** A ledger is self-attested by default — it proves this steward wrote this sequence, not that the decisions were right or that a second party saw them. The family already holds the shape of the answer elsewhere (a meter attestation signed by a key distinct from the receipt signer); `pool/1` has no analogue for the *decision* dimension yet. Stated in the spec rather than left for a reader to find. **Totals: 31 tests**, conformance §7.1–14, `check-spec-refs` 1614 citations / 16 standards, 40 lockfiles current, `cargo metadata --locked` 0, comparative-prose and vanity-metrics clean. ## Fourth slice: a witness distinct from the steward (SPEC §6.7–6.8) §8 named the limit the third slice left behind: **the steward signs its own decisions.** A ledger proves this steward wrote this sequence — not that the decisions were right, and not that anyone else saw them. A party able to rewrite the whole stream can rewrite a *consistent* one, because it holds the only key involved. A **witness** is a party holding a key the steward does not. It signs the head at a stated sequence, producing a `pool-witness/1` receipt: a statement that the chain reached exactly that head at exactly that length. A steward that later rewrites its history produces a different head, and the receipt stops covering it. - **Demotion, not refusal** (§6.8). An unwitnessed ledger is still a ledger, graded `SelfAttested` rather than rejected — the shape this family already uses where a figure resolves to no independent attestation. What an implementation may not do is report the two as the same thing. - **One thing is refused outright.** A witness key equal to the steward's is not a witness. `verify` returns `WitnessIsTheSteward` rather than reporting a witnessed ledger: a countersignature from the party being countersigned establishes nothing, and would be the most misleading possible reading of the word. - **Unverifiable and covers-a-different-ledger are distinct errors.** The second is the interesting one — it is what a rewritten history looks like from the verifier's side — and it carries both the head the witness signed and the head actually offered. ### Falsification, fourth slice | mutation | result | |---|---| | **W1** the steward may witness itself | **only** `the_steward_cannot_witness_itself` FAILED | | **W2** head/seq not compared — a rewrite stays covered | **only** `a_rewritten_history_is_no_longer_covered` FAILED | | **W3** the steward field not checked | **only** `a_receipt_for_another_steward_is_refused` FAILED | | **W4** self-attested reported as witnessed | **only** `self_attested_is_a_grade_not_an_error` FAILED | Each rule is pinned by exactly one test, and the headline test is the one the slice exists for: a steward rewrites its history, every surviving entry still verifies individually, and the witness's receipt no longer covers the head. ## Fifth slice: attributing a shared device's energy (SPEC §2.5) Serving several workloads from one physical accelerator is how a scheduler stops stranding capacity, and the memory fence that bounds each tenant is enforceable **because memory is partitioned**. Energy is not. A device counter reports one figure for the whole device and has no notion of which tenant caused which joule — so a per-tenant figure is a **split of a measurement, not a measurement**. Two rules follow, and `share` exists to make them hold rather than to be remembered: - **Provenance degrades on division.** A `HwShunt` device total becomes `ModelBased` the moment two or more occupants divide it, because the division is a model however good the counter was. It never improves — dividing an `Estimator` guess does not make it a model — and dividing by one is not a division, so a sole occupant keeps the device's own grade. An implementation **may not** report a shared device's per-tenant figure as `HwShunt`: that grade asserts a counter was read for *this* work, and on a shared device none was. - **The shares sum to the measurement, exactly.** Integer microjoules, largest remainder, ties broken on the claim id, arithmetic in `u128` so a wide device total cannot overflow the weighting. Conservation is what makes a tenant's figure reconcilable against the device that produced it at all. Attribution refuses a device with no occupants, or one whose occupants all carry zero weight: there is then no basis to divide on, and discarding the figure would lose energy the counter really read. The weight basis stays the operator's to choose (allotted memory, time slices, partitions) — `pool/1` requires only that one basis covers every occupant of one device and that the result is graded as the model it is. ### Falsification, fifth slice — and the decoration it found | mutation | result | |---|---| | **M1** no provenance degradation on division | `provenance_degrades_on_division_but_not_on_a_sole_occupant` + 2 FAILED | | **M2** no largest-remainder leftover pass | `the_shares_sum_to_the_measurement_exactly` + 2 FAILED | | **M3** no claim-id tie-break on equal remainders | **only** `the_leftover_is_distributed_deterministically` FAILED | | **M4** reconcile takes the best grade instead of the worst | **only** `reconciling_across_devices_takes_the_weakest_grade` FAILED | | **M5** no refusal of a weightless occupant set | **only** `a_device_with_nobody_on_it_and_a_weightless_set_are_refused` FAILED | M4 is the one worth reading. `reconcile` folds shares back to a total and keeps the **weaker** grade, because a sum is no better graded than its worst term. My first test for that fed it shares that all carried the *same* grade — so `min` and `max` were indistinguishable and the test pinned nothing. Reconciling **across** devices is where the grade actually matters (one device read by a counter, another only modelled), and that case is reachable, so it now has a test of its own. Note that the original same-grade test does **not** appear in M4's failure list: only the new one catches it. ## Verification after the fifth slice - `cargo check --workspace --all-targets --locked` (`pool-rs`) — exit **0** - `cargo test --workspace --locked` — exit **0**, **51** passed - `cargo clippy --workspace --all-targets --no-deps --locked` — clean - `cargo metadata --locked` — exit **0** - `check-spec-refs` — 1614 citations across 16 standards, all resolve - `check-lockfiles` — 40 lockfiles, all current - `check-tests-in-ci` — every test file is run by a job, and none can run empty - `check-comparative-prose` — clean (2332 files); `check-no-vanity-metrics` — clean (281 files) Conformance now runs §7.1–20. Still out of scope and named as such in §1.1 and §8: placement and topology-awareness, admission-time workload capture, idle reclamation, and a multi-cluster control plane. ## Sixth slice: a guess earns its way out (SPEC §2.6) Rule 1 refuses a guess the right to spend a neighbour's capacity. Reviewing the branch as a whole, that rule had no exit: **a tenant nobody has measured can never be measured, because it can never run against surplus.** As written it was a permanent penalty, not an incentive. The evidence was already in the ledger — every completed claim recorded what it actually consumed, at the grade of whatever read it. What was missing was *what the reservation was for*, so `Event::Admitted` now records the claim's **class**: the operator's assertion that two claims are the same kind of work. It goes in the entry rather than in a scheduler-side map because §6.1 makes the ledger the only durable state, and such a map is exactly the second store that can drift from it. `observe::derive` turns that history into an estimate under four rules: - **A guess is not evidence.** `Estimator` observations are excluded, not averaged in. Counting them would launder guesses into a grade asserting they were observed — and would make the exit circular, since a tenant would earn an observed estimate merely by having run. It earns one by being *measured*. - **A derived estimate is a model, never a reading.** Capped at `ModelBased` even where every observation came from a hardware counter, because a figure for the *next* run is a prediction. This is the same cap `share` applies to a divided device total, for the same reason. It is a ceiling, not a floor: modelled inputs give a modelled output. - **One run is not a sample.** Below the required count nothing is derived, and the counts are reported *separately* — observations of the class, and how many were evidence — because history that is all guesses needs a meter while little history needs patience. `required: 0` floors at one, so it can never mean "derive from nothing". The count is a policy value, and §2.6 says so rather than pretending `pool/1` knows the right number for a workload it has never seen. - **The absence of a class is not a class.** An empty class asserts nothing about comparability and is refused rather than pooling unlike work into one figure. The reservation takes the **largest** observed figure, not the mean: under-reserving is the failure rule 1 exists to prevent, and a mean under-reserves roughly half the time by construction. The median reported beside it is the lower middle value at an even count, so no reported figure is an average nothing measured. §2.6 states the limit — the maximum of a small sample underestimates the true tail, so this is better evidence than a guess, not a bound on what the work can cost. `ledger::observations` reads evidence back out of a verified stream. Only a matched `Released` yields an observation: an open admission is a reservation, not a result, and reserving a figure is not evidence that the work costs it. The stream's first pass is now extracted as `verified_events` and **shared** with `replay` — a second verification path is a second place for the checks to drift. ### Falsification, sixth slice | mutation | result | |---|---| | **O1** guesses counted as evidence | **only** `a_guess_is_not_evidence_however_much_of_it_there_is` FAILED | | **O2** derived estimate keeps the counter's grade | **only** `a_derived_estimate_is_never_graded_as_a_reading` FAILED | | **O3** no minimum evidence — one run is a sample | `one_run_is_not_a_sample` + 3 FAILED | | **O4** `required: 0` means derive from nothing | **only** `zero_required_still_needs_one_observation` FAILED | | **O5** the reservation takes the mean | `the_reservation_takes_the_largest_not_the_mean` + 2 FAILED | | **O6** an empty class pooled rather than refused | **only** `an_empty_class_is_refused_rather_than_pooled` FAILED | | **O7** observations read from an unverified stream | **only** `a_broken_stream_yields_no_evidence_at_all` FAILED | | **O8** an open admission counts as evidence | `an_open_admission_is_not_evidence` + 1 FAILED | O7 is worth a note on method: my first attempt at it **did not compile**, so it initially reported as surviving. A mutation that does not build tests nothing, and I rewrote it until it compiled before reading the result. The headline test runs end to end through a sealed ledger: three measured `train` claims leave the tenant inside its 5 MJ guarantee, the derivation produces 1 500 002 µJ from that history, and **the same number of microjoules** is then offered twice — queued as a guess, admitted by borrowing once it is derived from the ledger's own measured history. The figure comes out of the history, not out of the test. ### Totals after the sixth slice - `cargo check --workspace --all-targets --locked` — exit **0**; `cargo test --workspace --locked` — exit **0**, **63** passed - `cargo clippy --workspace --all-targets --no-deps --locked -- -D warnings` — clean; `cargo metadata --locked` — exit **0** - `check-spec-refs` — **1619** citations across 16 standards, all resolve - `check-lockfiles` — 40 lockfiles current; `check-tests-in-ci` — clean - `check-comparative-prose` and `check-no-vanity-metrics` — clean - Conformance now runs **§7.1–24** A side note on that spec-reference count. It had not moved across the two previous slices, which turned out to be because the guard only validates citations written as `SPEC §x.y` in `.rs` files, and this crate's doc comments used a bare `§`. They were therefore unchecked. Rewritten into the validated form, the count moved 1614 → 1619 and all resolve — the citations are now actually verified rather than merely present. Still out of scope and named as such in §1.1 and §8: placement and topology-awareness, idle reclamation, and a multi-cluster control plane. ## Seventh slice: reclaiming idle capacity without reclaiming on a missing number (SPEC §2.7) A workload holding accelerators while consuming almost no energy is holding capacity it is not using. A device ledger cannot see that — a held device is a held device, saturated or parked at an interactive prompt — so this is a question only an energy-denominated scheduler can ask. It is also **the most dangerous question in this standard**, because the obvious implementation destroys running work. `"this workload spent 0 µJ"` arrives identically from a workload that is genuinely parked and from a meter that is not there: ``` meter unavailable → 0 µJ → "idle" → reclaim → running work destroyed ``` That is this family's "unknown conflated with a value" defect in its most destructive form, so the module is built around refusing it rather than acquiring the refusal later: - **Unknown is not idle.** An `Estimator` figure cannot establish idleness whatever its value, and the grade is tested **before** the value meets any threshold — so no threshold setting lets a guess through, *including one high enough to catch everything*. A workload is reclaimed on evidence that it is idle, never on the absence of evidence that it is busy. - **Idleness is a rate over a stated window, never an instant.** Any workload is momentarily between steps. An observation shorter than the policy's window yields no verdict: startup is not idleness. An empty window yields **no rate** — `mean_uw()` returns `None`, not zero, because zero is the one value that reads as idle. - **Nothing held, nothing to reclaim.** - **Four verdicts, and only one permits reclamation.** A workload with no verdict is *absent* from `reclamation_order`, not ranked last in it: it is not a worse candidate than an idle workload, it is not a candidate. The check order is normative: nothing held, then the window, then the grade, then the rate. A later check never rescues an earlier refusal. ### Why there is no default threshold A parked accelerator still draws power. A threshold has to sit **above** the hardware's idle draw or nothing is ever idle, and **below** its working draw or everything is — and both are properties of the device, not of this standard. `pool/1` declines to guess them, so `IdlePolicy::default()` reclaims **nothing**. That is deliberate: the failure of an unconfigured destructive action should be a missed optimisation, not destroyed work. This is also not preemption. §5.2 ranks *who is spending badly under contention* by cost per completed unit; this asks *who is not spending at all*. A workload burning energy while completing nothing is thrashing, not idle, and §5.2 is the question that catches it. ### Falsification, seventh slice | mutation | result | |---|---| | **R1** a guess may establish idleness | `a_guess_of_zero_is_not_idle_though_a_reading_of_zero_is` + 2 FAILED | | **R2** grade tested *after* the value meets the threshold | `no_floor_setting_lets_a_guess_be_reclaimed` + 2 FAILED | | **R3** no minimum window — an instant is a verdict | `a_workload_younger_than_the_window_gets_no_verdict` + 1 FAILED | | **R4** an empty window is a rate of zero | **only** `an_empty_window_has_no_rate_rather_than_a_rate_of_zero` FAILED | | **R5** the unmeasured ranked last instead of absent | `the_unmeasured_are_absent_from_the_order_not_ranked_last` + 1 FAILED | | **R6** nothing-held treated as idle | **only** `a_workload_holding_nothing_is_not_a_candidate` FAILED | | **R7** the threshold is inclusive at its boundary | `the_floor_is_exclusive_at_its_boundary` + 1 FAILED | | **R8** the default policy reclaims rather than declining | **only** `the_default_policy_reclaims_nothing` FAILED | | **R9** the rate conversion overflows in `u64` | **only** `a_long_window_does_not_overflow_the_rate` FAILED | R2 is the one worth reading, because it is the mutation a careless refactor would actually produce: the grade check is still *present*, just moved below the threshold comparison. The behaviour is then correct for every workload except the ones that matter, and `no_floor_setting_lets_a_guess_be_reclaimed` is the test that pins the ordering rather than merely the presence of the check. As with O7 in the previous slice, R8's first form **did not compile** and so initially read as surviving. A mutation that does not build tests nothing; it was rewritten until it compiled before the result was read. The headline test is the discriminating pair: the same workload, the same devices, the same window, the same figure of zero — reclaimable when a counter read it, protected when nobody did. ### Totals, branch as it stands - `cargo check --workspace --all-targets --locked` — exit **0**; `cargo test --workspace --locked` — exit **0**, **74** passed - `cargo clippy --workspace --all-targets --no-deps --locked -- -D warnings` — clean - `cargo doc --no-deps` — **clean**; this slice also fixes three broken intra-doc links left by earlier ones (`WitnessReceipt`, the `derive` function/attribute ambiguity, and a public doc linking the private first pass) - `cargo metadata --locked` — exit **0** - `check-spec-refs` — **1621** citations across 16 standards, all resolve - `check-lockfiles` — 40 lockfiles current; `check-tests-in-ci` — clean - `check-comparative-prose` and `check-no-vanity-metrics` — clean - Conformance now runs **§7.1–28** Remaining out of scope, and named as such in §1.1 and §8: placement and topology-awareness, and a multi-cluster control plane. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 2 commits 2026-10-18 13:44:29 -04:00

pool: a standard for energy-accounted multi-tenant scheduling ... e559d020bb

Every deployed cluster scheduler denominates entitlement in devices —
whole accelerators, fractions of one, or a share of wall clock. Devices
are a proxy. The quantity actually consumed, paid for, and reported
under an energy-disclosure regime is energy, and a device-count quota
cannot express "this team may spend 40 MJ this window". So the budget
that matters is enforced nowhere and reconciled after the fact, if at
all. A survey of our own tree found the gap on both sides: `over-quota`
and `fair-share` appear in zero files, so we had no entitlement model
either.

`pool/1` denominates entitlement in microjoules AND devices, and
enforces both. Three rules carry it, and each needs the energy unit to
state at all:

1. **A guess may not spend a neighbour's capacity** (SPEC §4.4). A claim
   whose estimate is `Estimator` — nobody measured or modelled it — is
   admitted only inside its own guarantee, never against surplus
   borrowed from a sibling. A tenant may bear its own mis-estimate; a
   sibling's unspent joules are not the tenant's to gamble, and an
   unobserved estimate is the figure most likely to be wrong in the
   direction that overcommits.
2. **A gang is all-or-nothing in joules, not only in slots** (§2.4).
   Gang scheduling exists because half a distributed job is worse than
   none of it; the same holds for its energy. Admitting part of a gang
   against a budget that cannot finish the whole gang spends real joules
   and delivers nothing.
3. **Preemption follows evidence, then declaration** (§5.2). Workloads
   with a measured cost per COMPLETED unit are reclaimed worst-first.
   Workloads that have completed nothing are unmeasured, not wasteful,
   and fall back to declared priority.

Rule 3 is why the energy unit is load-bearing rather than decorative: it
ranks by cost per completed unit, which needs an outcome anchor. A
scheduler that knows only device-seconds cannot express the ordering.

Two disciplines from the family are built in from the start rather than
retrofitted:

- **Unknown is never an extreme.** `uj_per_completed_unit` returns
  `None` with no completed unit (§5.1) — not zero, which would make a
  new workload unreclaimable, and not infinity, which would make every
  new workload the first victim and destroy the very work that would
  have produced the measurement.
- **Pending and refused are different verdicts** (§4.5). A claim that
  cannot fit today queues; one that can never fit as written is denied,
  with the reason. Reporting the second as the first is how a queue
  fills with work that will never run.

Admission order is normative (§4), because a different order gives a
different verdict: impossibility first, then devices, then the
guarantee, then borrowing. Surplus is shared by weight against every
borrower under one parent, so a tenant cannot take it all by claiming
first, and never crosses roots.

Scope is deliberately narrow (§1.1, §7): `pool/1` decides admission and
preemption order. It does not place, meter or execute. A verdict is only
as sound as the figure handed to it, so the grade travels into the
verdict itself rather than sitting beside it.

10 tests, including the discriminating pair for rule 1 — the same claim
with the same surplus available, differing only in whether anyone
measured the estimate.

Co-Authored-By: Claude Opus 5 <[email protected]>

README: index pool/ among the standards ... 010fce3efe

The 16th standard directory needs its entry in the root index, in the
same shape as the others: what it decides, the three rules that need the
energy unit, and the honest scope line.

Co-Authored-By: Claude Opus 5 <[email protected]>

dcharlot added 1 commit 2026-10-18 15:14:08 -04:00

pool: usage is derived from a replayable ledger, not held in memory ... 8bc1d3bcdd

A scheduler that restarts with empty usage hands every tenant its whole
guarantee back and admits the next claim against a budget already spent.
A quota that refills on every deploy, crash or config reload is not a
quota. That defect has been found and fixed twice elsewhere in this
family this week — the JCP gateway refilled every grant on restart, and
the JouleHook gate's budget died with each per-action process — so
`pool/1` carries the replay from the start rather than acquiring it
after someone notices.

SPEC §6 and `ledger`:

- **Usage is derived, not stored** (§6.1). An implementation MUST rebuild
  every entitlement's usage by replaying its own ledger. The ledger is
  the durable state; there is no second store to drift from it.
- **Two events move it** (§6.2): `Admitted` reserves the estimate,
  `Released` lifts the hold and charges the **actual** figure in its
  place. A `Queue` or a `Deny` reserves nothing and is not recorded —
  that would make this a decision log, a different artifact with a
  different retention story.
- **The actual replaces the reservation, it is not added to it.** The
  hold bounded what could start; what the tenant spent is what it spent.
  An admission still open at the end of a stream keeps its reservation,
  because the capacity is still held.
- **An overrun is charged at what it cost** (§6.3), which is why the
  grade travels with every figure: the overrun is attributable.
- **The chain is verified before anything is derived** (§6.4), and a
  broken link is reported at its index rather than as a shorter history.
  A release naming a claim never admitted is refused at its index too:
  that is not a shorter history, it is a history that did not happen.

§6.5 states what this is not. The chain is tamper-evident as a
*sequence* — reorder, remove or alter an entry and the link breaks — but
it is **not authenticated**: nothing in v0.1 signs an entry, so a party
who can rewrite the whole stream can rewrite a consistent one. The
signed `pool/1` receipt (JCR-1 canonical CBOR under COSE_Sign1, as the
rest of the family seals) is a later slice, and until it exists §6.5
forbids an implementation from calling a ledger entry sealed, signed or
non-repudiable. The type is named `Ledger` and not `Receipt` for the
same reason: the word would claim the half that is missing.

Conformance gains four duties (§7.10–13). The honest limit moves to §8;
the two citations that pointed at the old number are updated, which
`check-spec-refs` would have caught either way.

The headline test is the discriminating pair the family now uses for
this defect: the same tenant with its guarantee spent, replayed — still
queued; not replayed — admitted. 17 tests.

Co-Authored-By: Claude Opus 5 <[email protected]>

dcharlot added 2 commits 2026-10-18 15:32:48 -04:00

pool: ledger entries are sealed pool/1 JCR-1 receipts ... 2bfaa00d22

§6.5 previously said what the ledger was not: tamper-evident as a
sequence, but unauthenticated, with the spec forbidding an
implementation from calling an entry sealed until the signed form
landed. It lands here, and §6.5 now says what the entry is.

Each entry is a `pool/1` JCR-1 receipt — canonical det-CBOR (RFC 8949
§4.2.1: length-first key order, shortest integer form, no floats, bools
or nulls) inside a COSE_Sign1 envelope (RFC 9052, EdDSA/Ed25519,
`alg = -8`), the steward's identifier as `kid`. That is the wire format
the rest of this family seals with, so a `pool/1` receipt verifies under
a tool that has never heard of Pool.

The JCR-1 primitives are **vendored**, which is what `mesh-receipt`
documents doing for the same reason: the reference library stays
self-contained while the bytes stay compatible with the family's
cross-language verifiers. Vendoring also kept the dependency honest —
`mesh-receipt` reaches into `jcp-schema` for the decoder, and taking
that dependency here would have contradicted the "self-contained"
rationale in the vendored header, so the decoder is written against
JCR-1's own grammar instead.

Two properties, checked separately because they fail separately:

- **Authenticity** — every entry verifies under the steward's key.
- **Sequence** — every `seq` is its position and every `prev` is the
  content address of the entry before it, from genesis.

Neither alone is enough, and the tests say so rather than assuming it.
`a_removed_entry_breaks_the_chain_though_each_entry_is_still_signed`
deletes an entry, asserts the survivors STILL verify individually, and
asserts the stream is refused anyway: signatures without a chain are a
bag of receipts, not a ledger.

§6.6 is new: canonicity is checked, not assumed. A signature covers
exact bytes, so a verifier that accepted a non-canonical encoding of the
same document would let two byte strings share one signature. The
decoder re-encodes what it decoded and refuses any difference.

The decoder is the one piece here that is new code rather than vendored,
so it carries its own tests: round-trip, a non-shortest integer refused,
map keys out of length-first order refused, trailing bytes refused,
duplicate keys refused, every truncation of a valid core refused, and
the types outside the core grammar (floats, bools, nulls, negative
integers, byte strings) refused.

§8 gains the limit this does NOT remove: the steward signs its own
decisions, so a ledger is self-attested by default — it proves this
steward wrote this sequence, not that the decisions were right or that a
second party saw them. The family already holds the shape of the answer
elsewhere, a meter attestation signed by a key distinct from the receipt
signer, and `pool/1` has no analogue for the decision dimension yet.
Said in the spec rather than left for a reader to discover.

Conformance §7.13 is rewritten and §7.14 added. 31 tests.

Co-Authored-By: Claude Opus 5 <[email protected]>

pool: two decoder tests assert the reason, not merely the refusal ... 471afd3c31

Falsification caught these: deleting the trailing-bytes guard and the
duplicate-key guard changed no test outcome. Both inputs are also
rejected by the canonicity re-encode check (§6.6) — a shorter or
smaller re-encoding never matches the input — so `is_err()` passed with
the guard gone and pinned nothing.

Each test now matches the specific `MalformedCore` reason, so the guard
it names is the thing under test. Re-falsified: deleting the
trailing-bytes guard now fails its test.

The guards stay rather than being deleted as redundant: they give a
precise reason where the canonicity check gives a general one, and a
verifier's diagnostics are part of what makes a refusal actionable.

Co-Authored-By: Claude Opus 5 <[email protected]>

dcharlot added 1 commit 2026-10-18 15:43:28 -04:00

pool: a witness distinct from the steward (SPEC §6.7-6.8) ... be6e44ef82

§8 recorded the limit the last slice did not remove: the steward signs
its own decisions, so a ledger establishes that THIS steward wrote THIS
sequence and nothing more. A steward able to rewrite its whole stream
can rewrite a consistent one, because it holds the only key involved.

A witness is a party holding a key the steward does not. It signs the
ledger's head at a stated length, producing a `pool-witness/1` receipt.
A steward that later rewrites its history produces a different head, and
the receipt stops covering it — which is what
`a_rewritten_history_is_no_longer_covered` asserts: the rewritten chain
is internally perfect, every entry still validly sealed by the same
steward key, and the witness's receipt refuses it anyway on the head.

Two design choices, both taken from shapes this family already uses:

- **Demotion, not refusal** (§6.8). An unwitnessed ledger is still a
  ledger: it is graded `self-attested` rather than rejected, as
  `joulecontract` grades a figure that resolves to no independent meter
  attestation. What an implementation MUST NOT do is report the two as
  the same thing. The reader weighs the difference; the standard's duty
  is to make it legible.
- **A witness key equal to the steward's is refused outright** (§6.7).
  A countersignature from the party being countersigned establishes
  nothing, and calling it witnessed would be the most misleading reading
  of the word available. This is the one case the module refuses rather
  than grades.

§6.8 also separates two failures that a single `is_err()` would blur: a
receipt that does not verify is a bad signature; a receipt that verifies
and covers a different head is what a rewritten history looks like from
outside. The error type names them apart and carries both heads, so the
diagnosis is in the value rather than in a log line.

§8 is rewritten rather than deleted, because the limit narrowed instead
of vanishing. Two things stay true: nothing compels a witness, so the
guarantee a reader holds depends on a deployment choice; and a witness
attests the SEQUENCE, not the decisions — it is evidence against
revision, not a second opinion on whether an admission was right.

Conformance gains §7.15-17. 39 tests, no warnings. Two pieces of my own
housekeeping went with it: a filler test that existed only to justify an
unused import is gone rather than kept to pad a count, and the vendored
`KidMismatch` fields are documented rather than having the lint silenced.

Co-Authored-By: Claude Opus 5 <[email protected]>

dcharlot added 1 commit 2026-10-18 19:09:09 -04:00

pool/1: attribute a shared device's energy, and grade the split honestly ... feb6b9a13d

A fractional accelerator's memory fence is enforceable because memory is
partitioned. Energy is not: a device counter reports one figure for the whole
device and cannot see tenants. A per-tenant figure from a shared device is
therefore a split of a measurement, not a measurement.

`share` encodes the two rules that follow (SPEC §2.5):

- Provenance degrades on division. A `HwShunt` device total becomes
  `ModelBased` the moment two or more occupants divide it, because the division
  is a model however good the counter was. It never improves — dividing an
  `Estimator` guess does not make it a model — and dividing by one is not a
  division, so a sole occupant keeps the device's grade. An implementation may
  not report a shared device's per-tenant figure as `HwShunt`: that grade
  asserts a counter was read for *this* work, and none was.

- The shares sum to the measurement exactly. Integer microjoules, largest
  remainder, ties broken on the claim id, arithmetic in u128 so a wide device
  total cannot overflow the weighting. Conservation is what makes a tenant's
  figure reconcilable against the device that produced it.

`attribute` refuses a device with no occupants or no weight: there is then no
basis to divide on, and discarding the figure would lose energy the counter
really read. `reconcile` folds shares back to a total and keeps the weaker
grade. `by_device` groups occupants so a caller attributes per device rather
than across the pool.

Conformance gains §7.18–20.

Co-Authored-By: Claude Opus 5 <[email protected]>

dcharlot added 1 commit 2026-10-18 19:21:17 -04:00

pool/1: a guess earns its way out (SPEC §2.6) ... 668ccc310e

SPEC §4.4 refuses a guess the right to spend a neighbour's capacity. That rule
is only fair if a guess can stop being one, and on its own it is a permanent
penalty with no exit: a tenant nobody has measured can never be measured,
because it can never run against surplus.

The evidence was already in the ledger — every completed claim recorded what it
actually consumed, at the grade of whatever read it. What was missing was *what
the reservation was for*, so `Event::Admitted` now records the claim's class.
It goes in the entry rather than in a scheduler-side map because §6.1 makes the
ledger the only durable state, and such a map is exactly the second store that
can drift from it.

`observe::derive` turns that history into an estimate under four rules:

- **A guess is not evidence.** `Estimator` observations are excluded, not
  averaged in. Counting them would launder guesses into a grade asserting they
  were observed, and would make the exit circular: a tenant earns an observed
  estimate by being measured, not by having run.
- **A derived estimate is a model, never a reading** — capped at `ModelBased`
  even where every observation came from a counter, because a figure for the
  *next* run is a prediction. The same cap `share` applies to a divided device
  total, for the same reason. A ceiling, not a floor.
- **One run is not a sample.** Below the required count nothing is derived and
  the counts are reported separately, because history that is all guesses needs
  a meter while little history needs patience. `required: 0` floors at one, so
  it can never mean "derive from nothing". The count is a policy value, not a
  derived one.
- **The absence of a class is not a class.** An empty class asserts nothing
  about comparability and is refused rather than pooling unlike work.

The reservation takes the largest observed figure, not the mean: under-reserving
is the failure §4.4 exists to prevent, and a mean under-reserves roughly half
the time by construction. The median is the lower middle value at an even
count, so no reported figure is an average nothing measured. §2.6 states the
limit — the maximum of a small sample underestimates the true tail, so this is
better evidence than a guess, not a bound.

`ledger::observations` reads evidence back out of a verified stream, and the
first pass is now extracted as `verified_events` and shared with `replay`: a
second verification path is a second place for the checks to drift. Only a
matched `Released` yields an observation — an open admission is a reservation,
not a result.

Conformance gains §7.21–24, and the spec citations in this crate are written in
the form `check-spec-refs` actually validates (1614 → 1619 citations).

Co-Authored-By: Claude Opus 5 <[email protected]>

dcharlot added 1 commit 2026-10-18 19:26:48 -04:00

pool/1: reclaim idle capacity without reclaiming on a missing number (SPEC §2.7) ... e5056334af

A workload holding accelerators while consuming almost no energy is holding
capacity it is not using. A device ledger cannot see that — a held device is a
held device, saturated or parked at a prompt — so idle reclamation is a question
only an energy-denominated scheduler can ask.

It is also the most dangerous question in this standard, because the obvious
implementation destroys running work. "This workload spent 0 µJ" arrives
identically from a parked workload and from a meter that is not there:

    meter unavailable → 0 µJ → "idle" → reclaim → running work destroyed

That is this family's "unknown conflated with a value" defect in its most
destructive form, so the module is built around refusing it:

- **Unknown is not idle.** An `Estimator` figure cannot establish idleness
  whatever its value, and the grade is tested **before** the value meets any
  threshold — so no threshold setting lets a guess through, including one high
  enough to catch everything. A workload is reclaimed on evidence that it is
  idle, never on the absence of evidence that it is busy.
- **Idleness is a rate over a stated window, never an instant.** Any workload
  is momentarily between steps. An observation shorter than the policy's window
  yields no verdict: startup is not idleness. An empty window yields no rate —
  `mean_uw()` returns `None`, not zero, because zero is the one value that
  reads as idle.
- **Nothing held, nothing to reclaim.**
- **Four verdicts, and only one permits reclamation.** A workload with no
  verdict is *absent* from `reclamation_order`, not ranked last in it: it is not
  a worse candidate than an idle workload, it is not a candidate.

Check order is normative: nothing held, then the window, then the grade, then
the rate. A later check never rescues an earlier refusal.

`IdlePolicy::default()` reclaims **nothing**. A threshold must sit above the
hardware's idle draw or nothing is ever idle, and below its working draw or
everything is, and both are properties of the device rather than of this
standard — so §2.7 declines to guess and makes the unconfigured failure "nobody
reclaimed" rather than "everybody reclaimed".

This is not preemption. §5.2 ranks who is spending badly under contention; this
asks who is not spending at all. A workload burning energy while completing
nothing is thrashing, not idle, and §5.2 is what catches it.

Conformance gains §7.25–28. Also fixes three broken intra-doc links left by
earlier slices (`WitnessReceipt`, the `derive` function/attribute ambiguity, and
a public doc linking the private first pass), so `cargo doc` is clean.

Co-Authored-By: Claude Opus 5 <[email protected]>

dcharlot merged commit f9e126a9b2 into main 2026-10-19 11:12:37 -04:00

dcharlot referenced this pull request from a commit 2026-10-19 11:12:38 -04:00

Merge pull request 'pool: a standard for energy-accounted multi-tenant scheduling' (#323) from feat/pool-energy-accounted-scheduling into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!323

No description provided.