2f3b000153
Decision models — state plus typed questions in, a calibrated probability per
allowed answer out, no text generated — went from nonexistent to a category
between 2026-09-15 and 2026-10-01, converging on three primitives (`noul`,
`choice`, `score`) and one schema. Every quantity in Mesh is a token count, and
a decision emits no tokens, so the spec could neither price one, rate one, nor
report one. Three defects, each checked in the code rather than inferred:
- `MeshReceipt::uj_per_accepted_token`, documented as "the wedge metric", is
`checked_div` by `tokens_accepted`. A decision's is 0 — measured
`"output_tokens": 0` on every request — so the headline efficiency figure
returned `None` for the whole category. The receipt sealed with no number.
- `estimate()` charged `prompt_tokens + max_tokens`, so a caller that left
`max_tokens` at a generation's default was priced for an essay the backend
will never write, and refused at a ceiling that would have covered the work.
- Nothing could express the dimension a decision's cost actually follows: the
schema it scores. One question over 20 options is more work than one over 2
at identical token counts.
Added `DecisionShape` on the request and `DecisionOutcome` on the outcome
(questions, options scored, and how much of the input was schema rather than
state), `CostProfile::with_option_cost` + `estimate_decision_uj`, and
`uj_per_decision` / `uj_per_option_scored` on the receipt. SPEC §3.6 states the
three MUSTs; §4's estimate formula and §6's field table are updated.
Additive throughout, following the pattern `CostProfile::calibration` already
set here: the new fields are `Option`, skipped when absent, and a declared
option cost of zero reproduces the token-only model exactly. So an existing
request's `request_id` is unchanged, a generation's signed core gains no keys,
and **the sealed conformance vectors in `conformance/v1/reference` still verify
byte-identically** — which is the real test that this is compatible, and it
passes. 52 tests, 0 failures, 0 clippy findings.
Two of the four new tests are the defects stated as invariants: a decision
receipt now has an efficiency figure where the token metric has none, and a
decision is not priced for tokens it never emits. Both were falsified by
reverting the fix — removing the `estimate()` branch fails two router tests,
making `uj_per_decision` read `tokens_accepted` fails two receipt tests.
`tools/joules-per-decision/` is the evidence: a harness that meters a real
Apache-2.0 decision model (Laya, ModernBERT-large 421M, via MLX) per fabric on
Apple silicon, and the numbers in §3.6 come from it. On an M5 Max a
three-question decision costs **0.104 J** — 0.0966 J on the GPU rail plus
0.0075 J of process CPU energy — at an 11.85 ms p50, which is ~35 mJ per
answered question. Idle baselines bracket the run and correct it by 0.08%, so
the figure is measurement, not modelling. The energy counters are read
unprivileged; the CPU-rail/GPU-rail asymmetry on macOS is noted there and is
the subject of follow-up work on `jcp-energy`.
The harness also reproduced a live defect in the category it measures: the
published Laya checkpoint ships a fitted temperature of 0.1006 for choice
questions with 11+ options, which sharpens logits roughly tenfold and reports
near-certainty on what may be a coin flip. A third-party porter found it by
reproducing upstream numerically and clamps it with a warning; the publisher
ships it. A calibrated probability is a claim, and this category ships claims
with no receipt — which is the argument for §3.6 in one line.
Co-Authored-By: Claude Opus 5 <[email protected]>
80 lines
3.6 KiB
Python
80 lines
3.6 KiB
Python
"""Joules per typed decision, metered with the same counter jcp-energy's rung 3 uses.
|
|
|
|
ABI constants are lifted from jcp-energy (sizeof/offsetof-verified against the
|
|
macOS 26, 26.5 and 27 SDKs), not re-derived here.
|
|
"""
|
|
import ctypes, os, time, statistics, json, warnings
|
|
|
|
FLAVOR, SIZE, ENERGY_NJ, PENERGY_NJ = 6, 464, 336, 344
|
|
_libc = ctypes.CDLL("/usr/lib/libSystem.B.dylib")
|
|
_buf = ctypes.create_string_buffer(SIZE)
|
|
_pid = os.getpid()
|
|
|
|
def read_nj():
|
|
rc = _libc.proc_pid_rusage(ctypes.c_int(_pid), ctypes.c_int(FLAVOR), ctypes.byref(_buf))
|
|
if rc != 0:
|
|
raise OSError(f"proc_pid_rusage failed: rc={rc}")
|
|
e = int.from_bytes(_buf.raw[ENERGY_NJ:ENERGY_NJ+8], "little")
|
|
p = int.from_bytes(_buf.raw[PENERGY_NJ:PENERGY_NJ+8], "little")
|
|
return e, p
|
|
|
|
def liveness(window_s=0.5, interval_s=0.02):
|
|
"""jcp-energy's rule: >=5 advancing intervals AND >=80% of them, over >=300ms."""
|
|
advanced = total = 0
|
|
prev, _ = read_nj(); t_end = time.perf_counter() + window_s
|
|
while time.perf_counter() < t_end:
|
|
t1 = time.perf_counter() + interval_s
|
|
x = 0
|
|
while time.perf_counter() < t1: # burn CPU
|
|
x += 1
|
|
cur, _ = read_nj(); total += 1
|
|
if cur > prev: advanced += 1
|
|
prev = cur
|
|
ok = advanced >= 5 and total > 0 and advanced / total >= 0.80
|
|
return ok, advanced, total
|
|
|
|
import laya_mlx as laya
|
|
with warnings.catch_warnings():
|
|
warnings.simplefilter("ignore")
|
|
agent = laya.load("aac6fef/laya-mlx", dtype="float16")
|
|
|
|
STATE = "I was billed twice. Please refund the duplicate today."
|
|
Q1 = {"refund": {"type":"noul","instructions":"Does the customer ask for money back?"}}
|
|
Q3 = {
|
|
"department": {"type":"choice","instructions":"Which team should handle this request?",
|
|
"criteria":{"billing":"invoices, payments, refunds","technical":"bugs and outages","sales":"new purchases"}},
|
|
"urgency": {"type":"score","instructions":"How urgent is this request?","criteria":["not urgent","soon","critical"]},
|
|
"refund": {"type":"noul","instructions":"Does the customer ask for money back?"}}
|
|
|
|
def bench(label, questions, n, device):
|
|
a = laya.load("aac6fef/laya-mlx", dtype="float16", device=device) if device else agent
|
|
for _ in range(3): a.predict(STATE, questions) # warm
|
|
e0, p0 = read_nj(); lat = []
|
|
t0 = time.perf_counter()
|
|
for _ in range(n):
|
|
t = time.perf_counter(); r = a.predict(STATE, questions); lat.append((time.perf_counter()-t)*1e3)
|
|
wall = time.perf_counter() - t0
|
|
e1, p1 = read_nj()
|
|
u = r["usage"]
|
|
return {
|
|
"label": label, "device": device or "default", "n": n,
|
|
"p50_ms": round(statistics.median(lat), 2),
|
|
"p95_ms": round(sorted(lat)[int(0.95*len(lat))-1], 2),
|
|
"wall_s": round(wall, 3),
|
|
"cpu_J_total": round((e1-e0)/1e9, 6),
|
|
"cpu_J_per_decision": round((e1-e0)/1e9/n, 6),
|
|
"pcore_J_per_decision": round((p1-p0)/1e9/n, 6),
|
|
"input_tokens": u["input_tokens"], "output_tokens": u["output_tokens"],
|
|
"state_tokens": u["state_tokens"],
|
|
}
|
|
|
|
ok, adv, tot = liveness()
|
|
print(json.dumps({"counter":"ri_energy_nj via proc_pid_rusage(RUSAGE_INFO_V6)",
|
|
"live": ok, "advancing_intervals": adv, "intervals": tot,
|
|
"rule": "jcp-energy rung 3: >=5 advancing AND >=80%, >=300ms"}, indent=2))
|
|
rows = []
|
|
for dev in ("cpu", "gpu"):
|
|
for lbl, qs, n in (("1 question (noul)", Q1, 30), ("3 questions (choice+score+noul)", Q3, 30)):
|
|
try: rows.append(bench(lbl, qs, n, dev))
|
|
except Exception as ex: rows.append({"label": lbl, "device": dev, "error": repr(ex)[:200]})
|
|
print(json.dumps(rows, indent=2))
|