740b6b21d2
Re-ran the two-pass L4 measurement on `real` after agent-joule cfe83c4. Every repeat closed at L0 for $0, including both multi-file changes that previously paid the cloud in full on every pass. The cascade now pays exactly once per distinct task regardless of how many files the fix touches, and $/resolved falls from $0.3176 to $0.2223. The comparison is stated narrowly on purpose. With every task repeated once and every repeat cached, the cascade makes half the cloud calls a stateless agent does, so at equal per-call prices it costs exactly half — by construction, not as a discovery. Against the earlier claude baseline the gap is only 21%, because this run's per-call prices were higher and the two figures come from different runs. The structural claim is the robust one; the percentage is not. Timing and local energy from this run are explicitly marked invalid. The host sat at load average 50-87 from unrelated work, and one row reports 90,155 J of local energy for a 25 s run — about 3.5 kW. Tier outcomes and dollar costs are not affected by CPU contention and are reported; nothing else from this run should be read. Co-Authored-By: Claude Opus 5 <[email protected]>
14 KiB
joule-bench results
Re-measured 2026-09-10 on Apple Silicon, after fixing three defects that made every earlier figure in this file wrong. The old numbers and what was wrong with them are kept at the bottom rather than deleted.
Resolution is graded by an executable test oracle: the task's tests must fail before the solver runs and pass after, with pristine test files restored first so a solver cannot pass by rewriting the test. Local energy is host-counter measured, idle-subtracted (idle calibrated per run at 36.9–38.4 W).
The mix suite (and what it is not)
Suite tasks/mix: four mechanical edits (rename, typo, constant, string) plus
one novel logic bug. One pass, L4 disabled, so this is purely local.
This suite is not a workload model, and it used to claim to be. It was called a "realistic operation mix" on the grounds that mechanical edits dominate real work. Measured against actual history, they do not. Classifying 1,679 source-touching commits across three real repositories:
joulesperbit open-standards agent-joule
multi-file 53.4% 66.0% 62.9%
single-file, large 37.8% 30.3% 27.8%
single-file, small (<=20 lines) 8.8% 3.7% 9.4%
single-file, <=10 lines 3.3% 2.3% —
strict one-token rename 0.0% 0.0% 0.0%
Not one commit in 1,679 was a single-file consistent one-token rename. On the
generous reading — any single-file change under ten lines — real work is
2.3-3.3% of that shape. mix is 80%. So the suite over-represents the
cascade's cheapest tier by roughly 25-40x, and a resolution figure from it
must not be read as a workload result.
What the measurement cannot see is repetition. Commit history records what changed, never how often someone asked for something similar, and L0 (cache) and L2 (retrieval) exist for exactly that. So this understates the local tiers' real-world reach by an unknown amount — but the correction is unmeasured, and pointing at it is not a defence of the 80%.
What the suite does establish is narrower and still worth having: when work is L1-shaped, it closes deterministically, sub-second, at energy below the measurement floor, and configuring a model does not change that.
solver resolved resol.% note
reference 5/5 100.0% applies each task's known-good patch — the ceiling
cascade 4/5 80.0% four closed at L1; the logic bug did not close
noop 0/5 0.0% touches nothing — the floor
The ceiling and floor are both where they should be, which is the first time that has been true. Per task, with no model configured:
task tier wall resolved
const L1 1457 ms yes
greeting L1 291 ms yes
rename L1 346 ms yes
typo L1 824 ms yes
logic — 415 ms no
Configuring a local model no longer taxes work that does not need it
The same suite with a 1.6 GB LFM2-2.6B-Q4 wired in as L3:
task tier wall energy resolved
const L1 518 ms 0 J~ yes
greeting L1 557 ms 0 J~ yes
rename L1 525 ms 17.9 J~ yes
typo L1 421 ms 0 J~ yes
logic — 193991 ms 1616 J no (L3 attempted, did not close)
The four L1 closes cost the same with a model configured as without one. That is the point of the cascade and it was not true a week ago: before the lazy-load fix these same four tasks cost 52–92 J and 4.7–10.5 s each, all of it spent loading a model none of them touched. Two causes — eager construction, and an L2-index embedding computed after every close, including deterministic ones, which forced the load anyway. Indexing now happens only when the model is already resident.
~ marks runs shorter than the counter's minimum sampling window. Their
resolution is valid; their energy is not comparable and the harness says so
rather than printing a fabricated number.
The one energy figure here that is properly measurable is the failure: escalating the novel logic bug to a 2.6B local model cost 1.6 kJ over 194 s and did not close it. That is a capability result, not a harness one.
Pay once, free forever — measured
The cascade's headline claim needs a close expensive enough to be worth caching.
The local tiers cannot produce one on this suite, so this run enables L4 (the
claude CLI) and takes two passes over tasks/mix.
pass task tier wall $ cloud J local J
1 const L1 205 ms $0.0000 0 21.0~
1 greeting L1 822 ms $0.0000 0 0.0~
1 logic L4 24 669 ms $0.3680 38 823 594.3
1 rename L1 552 ms $0.0000 0 0.0~
1 typo L1 832 ms $0.0000 0 13.0~
2 const L0 613 ms $0.0000 0 0.0~
2 greeting L0 897 ms $0.0000 0 7.9~
2 logic L0 582 ms $0.0000 0 0.0~
2 rename L0 814 ms $0.0000 0 12.2~
2 typo L0 733 ms $0.0000 0 15.4~
cascade 10/10 100.0% $0.0368/resolved 3 882 cloud J/resolved
The novel logic bug cost $0.3680, 24.7 s and 38.8 kJ of (token-estimated) cloud energy the first time. The second time it closed at L0 for $0, no cloud, in 582 ms — 42x faster, with the entire cost eliminated. One cloud call amortises across ten resolutions to $0.0368 each.
This is the first time this result has been produced with a working meter. Note what it is: a demonstration that the cache eliminates the cost of a repeated request, on a suite whose repetition rate is 100% by construction. It is not a claim about how often real work repeats, which this benchmark does not measure.
The frontier, both solvers, same suite and passes
claude (Claude Code 2.1.168) run over the identical suite and two passes:
solver resolved resol.% $/resolved cloud J/res local J/res wall/task
cascade 10/10 100.0% $0.0368 3 882 66.4 0.2-24.7 s
claude 10/10 100.0% $0.2809 35 830 220.6 13-29 s
Identical resolution. 7.6x cheaper, 9.2x less cloud energy, 3.3x less local
energy. The mechanism is visible per pass rather than inferred: claude pays
full price on the second pass exactly as on the first ($0.2688-$0.2791 against
$0.2525-$0.3534), because a stateless cloud agent has nothing to amortise. The
cascade's entire second pass closed at L0 for $0.
Latency separates them further on the tiers that close locally: L0/L1 closes run 0.2-0.9 s against 13-29 s for a cloud round trip, though the cascade's own L4 escalation costs 24.7 s, which is the same round trip plus local overhead.
One prior belief corrected. It was recorded here that a cloud agent spends "~0 local joules" because its compute happens in the vendor's datacenter. With a working meter it spends 220.6 J per resolved task locally — the CLI process, orchestration and file I/O across a 13-29 s run are not free. The old figure was an artefact of the broken meter, not a property of cloud agents.
What this multiple is and is not
It is a measured result for this suite, which the section above shows is 80% L1-shaped against 2-3% in real commit history, and which repeats every task exactly once. Both properties favour the cascade, by construction. A workload with less repetition and more multi-file work would move the multiple toward 1, and nothing here bounds how far.
The real suite: what the local tiers do on representative work
mix over-represents L1-shaped work by 25-40x, so tasks/real was built to the
measured distribution instead: six tasks, three whose fix spans two source files,
two single-file rewrites of real logic, one small single-file fix. Ordinary bugs
— a sliding window that never prunes, a parser folding operators left to right, a
config loader silently accepting misspelled keys, interval merging that assumes
sorted input, a retry loop that retries unretryable failures, a page count that
drops the last partial page.
The suite is sound: reference 6/6, noop 0/6.
solver resolved resol.%
reference 6/6 100.0%
noop 0/6 0.0%
cascade 0/6 0.0% (local tiers only, no L4)
The local tiers close none of it. Against 4/5 on mix. None of these bugs is
a substitution, a cached repeat, or a paraphrase of something already solved, so
L0, L1 and L2 have nothing to offer and every task falls through. With L4
disabled they simply fail, in under a second each.
This is the result the composition analysis predicted, and it bounds the
cascade's local claim precisely: the cheap tiers address a real but narrow class
of work, and that class is not what fills a commit log. The 7.6x advantage
measured on mix comes from that suite's composition, not from a general
property.
The 0/6 was checked against my own mistake before being believed. The first
version of these tasks had no target field, so the cascade ran with an empty
document path and its single-file tiers had nothing to act on — a handicap I had
introduced, not a property of the cascade. With target set on the four tasks
whose fix really is single-file (the other two span two files, which is the point
of including them), the result is unchanged: still 0/6. These bugs need
reasoning — prune the list, reorder precedence, reject unknown keys, sort before
merging, tell permanent failures from transient, round the division up — and none
is a substitution, a cached repeat, or a paraphrase.
Amortisation: single-file first, then multi-file
Before the fix. Two passes, L4 enabled. Every single-file close came back at
L0 for free on the repeat; neither multi-file close did — both paid the cloud in
full again. The cascade resolved 11/12 at $0.3176 per resolved task, no
cheaper than raw claude.
The cause was not the open standard, whose Edit has always carried its own path.
The L4 capture read only the target file, so a fix spanning two files was seen as
no change and never recorded; target-less requests were keyed on an empty path;
and replay ignored edit.path altogether. Fixed in agent-joule cfe83c4.
After the fix. The same two-pass run:
pass task shape tier $
1 config-defaults MULTI-FILE L4 $0.5943
1 interval-merge single-file L4 $0.3927
1 pagination single-file L4 $0.4029
1 precedence single-file L4 $0.4417
1 rate-window MULTI-FILE L4 $0.3901
1 retry-backoff single-file L4 $0.4453
2 config-defaults MULTI-FILE L0 $0.0000
2 interval-merge single-file L0 $0.0000
2 pagination single-file L0 $0.0000
2 precedence single-file L0 $0.0000
2 rate-window MULTI-FILE L0 $0.0000
2 retry-backoff single-file L0 $0.0000
cascade 12/12 100.0% $0.2223/resolved
All six repeats closed at L0, both multi-file changes included. The cascade now pays exactly once per distinct task, whatever the shape of the change.
Stated precisely, because the comparison is easy to overstate. With every task
repeated once and every repeat cached, the cascade makes half the cloud calls a
stateless agent does — so at equal per-call prices it costs exactly half, by
construction. Against the claude baseline measured earlier ($0.2809) the gap is
only 21%, because this run's per-call prices were higher ($0.39-$0.59 against
$0.25-$0.35 then); per-call cost moves by more than 2x between runs, and those two
figures come from different runs. The structural claim is the robust one.
Timing and local energy from this run are not valid. Load average was 50-87
throughout, from unrelated work on the host. The clearest symptom is rate-window
pass one reporting 90,155 J of local energy for a 25 s run — about 3.5 kW. Tier
outcomes and dollar costs do not depend on CPU contention and are unaffected;
everything else in this run should be ignored.
What is not measured here
The agent loop. joule-coded agent exists and joule-bench has an agent
solver, but the on-device runtime currently generates at 1.73 tokens/sec on
this model — roughly twenty times slower than the hardware should manage. An
agent-loop row today would measure the runtime, not the harness, so none is
published. Enabling the loader's Metal decode driver (jouleclaw-runtime/fabric)
was measured and is worse: 1.18 tok/s and 48.1 J/token against 1.73 tok/s and
33.8 J/token on the reference path.
Anything beyond nine tasks. mix is 5, smoke 3, cascade 1. These suites
show the machinery works; they are far too small to support a comparative claim,
and as measured above their composition is not representative either.
Any neutral third-party harness. We have never been scored on one.
Superseded: the figures published before 2026-09-10
They reported cascade 10/10 100% and local energy ~0 on this suite. Three
defects, each fatal to a different column:
- Every energy reading measured the meter. One
read()on the Apple model-based counter takes ~840 ms and reports the energy of its own sampling window; the meter bracketed each task between two of them. A 0 ms task read as 34 J. The counter is now sampled on a background thread. - Idle draw was calibrated at about nine times reality. The calibration slept 300 ms between two of those 840 ms reads, then divided an energy delta covering ~2 s by 0.3 s — reporting 352.2 W idle for a Mac. It is 38 W. That inflated figure was subtracted from every task, which is the real reason the old local-energy column read ~0.
- Resolution was under-reported on every Python task. The baseline run left a
__pycache__and the graded run imported the stale bytecode, so a correct fix still failed.reference— the ceiling — scored 4/5.
Nothing about the harness's design was wrong; it could simply not measure. Both the energy and the resolution columns were unusable, in opposite directions.