pool: a standard for energy-accounted multi-tenant scheduling #323

Merged

dcharlot merged 9 commits from feat/pool-energy-accounted-scheduling into main 2026-10-19 11:12:37 -04:00

Conversation 0 Commits 9 Files changed 17 +4967

2026-10-18

Transaction Science

pool/1: reclaim idle capacity without reclaiming on a missing number (SPEC §2.7) ...

A workload holding accelerators while consuming almost no energy is holding
capacity it is not using. A device ledger cannot see that — a held device is a
held device, saturated or parked at a prompt — so idle reclamation is a question
only an energy-denominated scheduler can ask.

It is also the most dangerous question in this standard, because the obvious
implementation destroys running work. "This workload spent 0 µJ" arrives
identically from a parked workload and from a meter that is not there:

    meter unavailable → 0 µJ → "idle" → reclaim → running work destroyed

That is this family's "unknown conflated with a value" defect in its most
destructive form, so the module is built around refusing it:

- **Unknown is not idle.** An `Estimator` figure cannot establish idleness
  whatever its value, and the grade is tested **before** the value meets any
  threshold — so no threshold setting lets a guess through, including one high
  enough to catch everything. A workload is reclaimed on evidence that it is
  idle, never on the absence of evidence that it is busy.
- **Idleness is a rate over a stated window, never an instant.** Any workload
  is momentarily between steps. An observation shorter than the policy's window
  yields no verdict: startup is not idleness. An empty window yields no rate —
  `mean_uw()` returns `None`, not zero, because zero is the one value that
  reads as idle.
- **Nothing held, nothing to reclaim.**
- **Four verdicts, and only one permits reclamation.** A workload with no
  verdict is *absent* from `reclamation_order`, not ranked last in it: it is not
  a worse candidate than an idle workload, it is not a candidate.

Check order is normative: nothing held, then the window, then the grade, then
the rate. A later check never rescues an earlier refusal.

`IdlePolicy::default()` reclaims **nothing**. A threshold must sit above the
hardware's idle draw or nothing is ever idle, and below its working draw or
everything is, and both are properties of the device rather than of this
standard — so §2.7 declines to guess and makes the unconfigured failure "nobody
reclaimed" rather than "everybody reclaimed".

This is not preemption. §5.2 ranks who is spending badly under contention; this
asks who is not spending at all. A workload burning energy while completing
nothing is thrashing, not idle, and §5.2 is what catches it.

Conformance gains §7.25–28. Also fixes three broken intra-doc links left by
earlier slices (`WitnessReceipt`, the `derive` function/attribute ambiguity, and
a public doc linking the private first pass), so `cargo doc` is clean.

Co-Authored-By: Claude Opus 5 <[email protected]>

e5056334af

• View at this point in history

2026-10-18 19:25:23 -04:00

Transaction Science

pool/1: a guess earns its way out (SPEC §2.6) ...

SPEC §4.4 refuses a guess the right to spend a neighbour's capacity. That rule
is only fair if a guess can stop being one, and on its own it is a permanent
penalty with no exit: a tenant nobody has measured can never be measured,
because it can never run against surplus.

The evidence was already in the ledger — every completed claim recorded what it
actually consumed, at the grade of whatever read it. What was missing was *what
the reservation was for*, so `Event::Admitted` now records the claim's class.
It goes in the entry rather than in a scheduler-side map because §6.1 makes the
ledger the only durable state, and such a map is exactly the second store that
can drift from it.

`observe::derive` turns that history into an estimate under four rules:

- **A guess is not evidence.** `Estimator` observations are excluded, not
  averaged in. Counting them would launder guesses into a grade asserting they
  were observed, and would make the exit circular: a tenant earns an observed
  estimate by being measured, not by having run.
- **A derived estimate is a model, never a reading** — capped at `ModelBased`
  even where every observation came from a counter, because a figure for the
  *next* run is a prediction. The same cap `share` applies to a divided device
  total, for the same reason. A ceiling, not a floor.
- **One run is not a sample.** Below the required count nothing is derived and
  the counts are reported separately, because history that is all guesses needs
  a meter while little history needs patience. `required: 0` floors at one, so
  it can never mean "derive from nothing". The count is a policy value, not a
  derived one.
- **The absence of a class is not a class.** An empty class asserts nothing
  about comparability and is refused rather than pooling unlike work.

The reservation takes the largest observed figure, not the mean: under-reserving
is the failure §4.4 exists to prevent, and a mean under-reserves roughly half
the time by construction. The median is the lower middle value at an even
count, so no reported figure is an average nothing measured. §2.6 states the
limit — the maximum of a small sample underestimates the true tail, so this is
better evidence than a guess, not a bound.

`ledger::observations` reads evidence back out of a verified stream, and the
first pass is now extracted as `verified_events` and shared with `replay`: a
second verification path is a second place for the checks to drift. Only a
matched `Released` yields an observation — an open admission is a reservation,
not a result.

Conformance gains §7.21–24, and the spec citations in this crate are written in
the form `check-spec-refs` actually validates (1614 → 1619 citations).

Co-Authored-By: Claude Opus 5 <[email protected]>

668ccc310e

• View at this point in history

2026-10-18 19:20:03 -04:00

Transaction Science

pool/1: attribute a shared device's energy, and grade the split honestly ...

A fractional accelerator's memory fence is enforceable because memory is
partitioned. Energy is not: a device counter reports one figure for the whole
device and cannot see tenants. A per-tenant figure from a shared device is
therefore a split of a measurement, not a measurement.

`share` encodes the two rules that follow (SPEC §2.5):

- Provenance degrades on division. A `HwShunt` device total becomes
  `ModelBased` the moment two or more occupants divide it, because the division
  is a model however good the counter was. It never improves — dividing an
  `Estimator` guess does not make it a model — and dividing by one is not a
  division, so a sole occupant keeps the device's grade. An implementation may
  not report a shared device's per-tenant figure as `HwShunt`: that grade
  asserts a counter was read for *this* work, and none was.

- The shares sum to the measurement exactly. Integer microjoules, largest
  remainder, ties broken on the claim id, arithmetic in u128 so a wide device
  total cannot overflow the weighting. Conservation is what makes a tenant's
  figure reconcilable against the device that produced it.

`attribute` refuses a device with no occupants or no weight: there is then no
basis to divide on, and discarding the figure would lose energy the counter
really read. `reconcile` folds shares back to a total and keeps the weaker
grade. `by_device` groups occupants so a caller attributes per device rather
than across the pool.

Conformance gains §7.18–20.

Co-Authored-By: Claude Opus 5 <[email protected]>

feb6b9a13d

• View at this point in history

2026-10-18 19:07:36 -04:00

Transaction Science

pool: a witness distinct from the steward (SPEC §6.7-6.8) ...

§8 recorded the limit the last slice did not remove: the steward signs
its own decisions, so a ledger establishes that THIS steward wrote THIS
sequence and nothing more. A steward able to rewrite its whole stream
can rewrite a consistent one, because it holds the only key involved.

A witness is a party holding a key the steward does not. It signs the
ledger's head at a stated length, producing a `pool-witness/1` receipt.
A steward that later rewrites its history produces a different head, and
the receipt stops covering it — which is what
`a_rewritten_history_is_no_longer_covered` asserts: the rewritten chain
is internally perfect, every entry still validly sealed by the same
steward key, and the witness's receipt refuses it anyway on the head.

Two design choices, both taken from shapes this family already uses:

- **Demotion, not refusal** (§6.8). An unwitnessed ledger is still a
  ledger: it is graded `self-attested` rather than rejected, as
  `joulecontract` grades a figure that resolves to no independent meter
  attestation. What an implementation MUST NOT do is report the two as
  the same thing. The reader weighs the difference; the standard's duty
  is to make it legible.
- **A witness key equal to the steward's is refused outright** (§6.7).
  A countersignature from the party being countersigned establishes
  nothing, and calling it witnessed would be the most misleading reading
  of the word available. This is the one case the module refuses rather
  than grades.

§6.8 also separates two failures that a single `is_err()` would blur: a
receipt that does not verify is a bad signature; a receipt that verifies
and covers a different head is what a rewritten history looks like from
outside. The error type names them apart and carries both heads, so the
diagnosis is in the value rather than in a log line.

§8 is rewritten rather than deleted, because the limit narrowed instead
of vanishing. Two things stay true: nothing compels a witness, so the
guarantee a reader holds depends on a deployment choice; and a witness
attests the SEQUENCE, not the decisions — it is evidence against
revision, not a second opinion on whether an admission was right.

Conformance gains §7.15-17. 39 tests, no warnings. Two pieces of my own
housekeeping went with it: a filler test that existed only to justify an
unused import is gone rather than kept to pad a count, and the vendored
`KidMismatch` fields are documented rather than having the lint silenced.

Co-Authored-By: Claude Opus 5 <[email protected]>

be6e44ef82

• View at this point in history

2026-10-18 15:42:36 -04:00

Transaction Science

pool: two decoder tests assert the reason, not merely the refusal ...

Falsification caught these: deleting the trailing-bytes guard and the
duplicate-key guard changed no test outcome. Both inputs are also
rejected by the canonicity re-encode check (§6.6) — a shorter or
smaller re-encoding never matches the input — so `is_err()` passed with
the guard gone and pinned nothing.

Each test now matches the specific `MalformedCore` reason, so the guard
it names is the thing under test. Re-falsified: deleting the
trailing-bytes guard now fails its test.

The guards stay rather than being deleted as redundant: they give a
precise reason where the canonicity check gives a general one, and a
verifier's diagnostics are part of what makes a refusal actionable.

Co-Authored-By: Claude Opus 5 <[email protected]>

471afd3c31

• View at this point in history

2026-10-18 15:32:00 -04:00

Transaction Science

pool: ledger entries are sealed pool/1 JCR-1 receipts ...

§6.5 previously said what the ledger was not: tamper-evident as a
sequence, but unauthenticated, with the spec forbidding an
implementation from calling an entry sealed until the signed form
landed. It lands here, and §6.5 now says what the entry is.

Each entry is a `pool/1` JCR-1 receipt — canonical det-CBOR (RFC 8949
§4.2.1: length-first key order, shortest integer form, no floats, bools
or nulls) inside a COSE_Sign1 envelope (RFC 9052, EdDSA/Ed25519,
`alg = -8`), the steward's identifier as `kid`. That is the wire format
the rest of this family seals with, so a `pool/1` receipt verifies under
a tool that has never heard of Pool.

The JCR-1 primitives are **vendored**, which is what `mesh-receipt`
documents doing for the same reason: the reference library stays
self-contained while the bytes stay compatible with the family's
cross-language verifiers. Vendoring also kept the dependency honest —
`mesh-receipt` reaches into `jcp-schema` for the decoder, and taking
that dependency here would have contradicted the "self-contained"
rationale in the vendored header, so the decoder is written against
JCR-1's own grammar instead.

Two properties, checked separately because they fail separately:

- **Authenticity** — every entry verifies under the steward's key.
- **Sequence** — every `seq` is its position and every `prev` is the
  content address of the entry before it, from genesis.

Neither alone is enough, and the tests say so rather than assuming it.
`a_removed_entry_breaks_the_chain_though_each_entry_is_still_signed`
deletes an entry, asserts the survivors STILL verify individually, and
asserts the stream is refused anyway: signatures without a chain are a
bag of receipts, not a ledger.

§6.6 is new: canonicity is checked, not assumed. A signature covers
exact bytes, so a verifier that accepted a non-canonical encoding of the
same document would let two byte strings share one signature. The
decoder re-encodes what it decoded and refuses any difference.

The decoder is the one piece here that is new code rather than vendored,
so it carries its own tests: round-trip, a non-shortest integer refused,
map keys out of length-first order refused, trailing bytes refused,
duplicate keys refused, every truncation of a valid core refused, and
the types outside the core grammar (floats, bools, nulls, negative
integers, byte strings) refused.

§8 gains the limit this does NOT remove: the steward signs its own
decisions, so a ledger is self-attested by default — it proves this
steward wrote this sequence, not that the decisions were right or that a
second party saw them. The family already holds the shape of the answer
elsewhere, a meter attestation signed by a key distinct from the receipt
signer, and `pool/1` has no analogue for the decision dimension yet.
Said in the spec rather than left for a reader to discover.

Conformance §7.13 is rewritten and §7.14 added. 31 tests.

Co-Authored-By: Claude Opus 5 <[email protected]>

2bfaa00d22

• View at this point in history

2026-10-18 15:30:07 -04:00

Transaction Science

pool: usage is derived from a replayable ledger, not held in memory ...

A scheduler that restarts with empty usage hands every tenant its whole
guarantee back and admits the next claim against a budget already spent.
A quota that refills on every deploy, crash or config reload is not a
quota. That defect has been found and fixed twice elsewhere in this
family this week — the JCP gateway refilled every grant on restart, and
the JouleHook gate's budget died with each per-action process — so
`pool/1` carries the replay from the start rather than acquiring it
after someone notices.

SPEC §6 and `ledger`:

- **Usage is derived, not stored** (§6.1). An implementation MUST rebuild
  every entitlement's usage by replaying its own ledger. The ledger is
  the durable state; there is no second store to drift from it.
- **Two events move it** (§6.2): `Admitted` reserves the estimate,
  `Released` lifts the hold and charges the **actual** figure in its
  place. A `Queue` or a `Deny` reserves nothing and is not recorded —
  that would make this a decision log, a different artifact with a
  different retention story.
- **The actual replaces the reservation, it is not added to it.** The
  hold bounded what could start; what the tenant spent is what it spent.
  An admission still open at the end of a stream keeps its reservation,
  because the capacity is still held.
- **An overrun is charged at what it cost** (§6.3), which is why the
  grade travels with every figure: the overrun is attributable.
- **The chain is verified before anything is derived** (§6.4), and a
  broken link is reported at its index rather than as a shorter history.
  A release naming a claim never admitted is refused at its index too:
  that is not a shorter history, it is a history that did not happen.

§6.5 states what this is not. The chain is tamper-evident as a
*sequence* — reorder, remove or alter an entry and the link breaks — but
it is **not authenticated**: nothing in v0.1 signs an entry, so a party
who can rewrite the whole stream can rewrite a consistent one. The
signed `pool/1` receipt (JCR-1 canonical CBOR under COSE_Sign1, as the
rest of the family seals) is a later slice, and until it exists §6.5
forbids an implementation from calling a ledger entry sealed, signed or
non-repudiable. The type is named `Ledger` and not `Receipt` for the
same reason: the word would claim the half that is missing.

Conformance gains four duties (§7.10–13). The honest limit moves to §8;
the two citations that pointed at the old number are updated, which
`check-spec-refs` would have caught either way.

The headline test is the discriminating pair the family now uses for
this defect: the same tenant with its guarantee spent, replayed — still
queued; not replayed — admitted. 17 tests.

Co-Authored-By: Claude Opus 5 <[email protected]>

8bc1d3bcdd

• View at this point in history

2026-10-18 15:12:59 -04:00

Transaction Science

README: index pool/ among the standards ...

The 16th standard directory needs its entry in the root index, in the
same shape as the others: what it decides, the three rules that need the
energy unit, and the honest scope line.

Co-Authored-By: Claude Opus 5 <[email protected]>

010fce3efe

• View at this point in history

2026-10-18 13:43:55 -04:00

Transaction Science

pool: a standard for energy-accounted multi-tenant scheduling ...

Every deployed cluster scheduler denominates entitlement in devices —
whole accelerators, fractions of one, or a share of wall clock. Devices
are a proxy. The quantity actually consumed, paid for, and reported
under an energy-disclosure regime is energy, and a device-count quota
cannot express "this team may spend 40 MJ this window". So the budget
that matters is enforced nowhere and reconciled after the fact, if at
all. A survey of our own tree found the gap on both sides: `over-quota`
and `fair-share` appear in zero files, so we had no entitlement model
either.

`pool/1` denominates entitlement in microjoules AND devices, and
enforces both. Three rules carry it, and each needs the energy unit to
state at all:

1. **A guess may not spend a neighbour's capacity** (SPEC §4.4). A claim
   whose estimate is `Estimator` — nobody measured or modelled it — is
   admitted only inside its own guarantee, never against surplus
   borrowed from a sibling. A tenant may bear its own mis-estimate; a
   sibling's unspent joules are not the tenant's to gamble, and an
   unobserved estimate is the figure most likely to be wrong in the
   direction that overcommits.
2. **A gang is all-or-nothing in joules, not only in slots** (§2.4).
   Gang scheduling exists because half a distributed job is worse than
   none of it; the same holds for its energy. Admitting part of a gang
   against a budget that cannot finish the whole gang spends real joules
   and delivers nothing.
3. **Preemption follows evidence, then declaration** (§5.2). Workloads
   with a measured cost per COMPLETED unit are reclaimed worst-first.
   Workloads that have completed nothing are unmeasured, not wasteful,
   and fall back to declared priority.

Rule 3 is why the energy unit is load-bearing rather than decorative: it
ranks by cost per completed unit, which needs an outcome anchor. A
scheduler that knows only device-seconds cannot express the ordering.

Two disciplines from the family are built in from the start rather than
retrofitted:

- **Unknown is never an extreme.** `uj_per_completed_unit` returns
  `None` with no completed unit (§5.1) — not zero, which would make a
  new workload unreclaimable, and not infinity, which would make every
  new workload the first victim and destroy the very work that would
  have produced the measurement.
- **Pending and refused are different verdicts** (§4.5). A claim that
  cannot fit today queues; one that can never fit as written is denied,
  with the reason. Reporting the second as the first is how a queue
  fills with work that will never run.

Admission order is normative (§4), because a different order gives a
different verdict: impossibility first, then devices, then the
guarantee, then borrowing. Surplus is shared by weight against every
borrower under one parent, so a tenant cannot take it all by claiming
first, and never crosses roots.

Scope is deliberately narrow (§1.1, §7): `pool/1` decides admission and
preemption order. It does not place, meter or execute. A verdict is only
as sound as the figure handed to it, so the grade travels into the
verdict itself rather than sitting beside it.

10 tests, including the discriminating pair for rule 1 — the same claim
with the same surplus available, differing only in whether anyone
measured the estimate.

Co-Authored-By: Claude Opus 5 <[email protected]>

e559d020bb

• View at this point in history

2026-10-18 13:42:26 -04:00