# joule-bench results Re-measured 2026-09-10 on Apple Silicon, after fixing three defects that made every earlier figure in this file wrong. The old numbers and what was wrong with them are kept at the bottom rather than deleted. Resolution is graded by an executable test oracle: the task's tests must fail before the solver runs and pass after, with pristine test files restored first so a solver cannot pass by rewriting the test. Local energy is host-counter measured, idle-subtracted (idle calibrated per run at **36.9–38.4 W**). ## The `mix` suite (and what it is not) Suite `tasks/mix`: four mechanical edits (rename, typo, constant, string) plus one novel logic bug. One pass, L4 disabled, so this is **purely local**. **This suite is not a workload model, and it used to claim to be.** It was called a "realistic operation mix" on the grounds that mechanical edits dominate real work. Measured against actual history, they do not. Classifying 1,679 source-touching commits across three real repositories: ``` joulesperbit open-standards agent-joule multi-file 53.4% 66.0% 62.9% single-file, large 37.8% 30.3% 27.8% single-file, small (<=20 lines) 8.8% 3.7% 9.4% single-file, <=10 lines 3.3% 2.3% — strict one-token rename 0.0% 0.0% 0.0% ``` Not one commit in 1,679 was a single-file consistent one-token rename. On the generous reading — any single-file change under ten lines — real work is **2.3-3.3%** of that shape. `mix` is **80%**. So the suite over-represents the cascade's cheapest tier by roughly **25-40x**, and a resolution figure from it must not be read as a workload result. What the measurement cannot see is repetition. Commit history records what changed, never how often someone asked for something similar, and L0 (cache) and L2 (retrieval) exist for exactly that. So this understates the local tiers' real-world reach by an unknown amount — but the correction is unmeasured, and pointing at it is not a defence of the 80%. What the suite does establish is narrower and still worth having: **when work is L1-shaped, it closes deterministically, sub-second, at energy below the measurement floor, and configuring a model does not change that.** ``` solver resolved resol.% note reference 5/5 100.0% applies each task's known-good patch — the ceiling cascade 4/5 80.0% four closed at L1; the logic bug did not close noop 0/5 0.0% touches nothing — the floor ``` The ceiling and floor are both where they should be, which is the first time that has been true. Per task, with no model configured: ``` task tier wall resolved const L1 1457 ms yes greeting L1 291 ms yes rename L1 346 ms yes typo L1 824 ms yes logic — 415 ms no ``` ## Configuring a local model no longer taxes work that does not need it The same suite with a 1.6 GB LFM2-2.6B-Q4 wired in as L3: ``` task tier wall energy resolved const L1 518 ms 0 J~ yes greeting L1 557 ms 0 J~ yes rename L1 525 ms 17.9 J~ yes typo L1 421 ms 0 J~ yes logic — 193991 ms 1616 J no (L3 attempted, did not close) ``` The four L1 closes cost the same with a model configured as without one. That is the point of the cascade and it was **not** true a week ago: before the lazy-load fix these same four tasks cost **52–92 J and 4.7–10.5 s each**, all of it spent loading a model none of them touched. Two causes — eager construction, and an L2-index embedding computed after *every* close, including deterministic ones, which forced the load anyway. Indexing now happens only when the model is already resident. `~` marks runs shorter than the counter's minimum sampling window. Their resolution is valid; their energy is not comparable and the harness says so rather than printing a fabricated number. The one energy figure here that is properly measurable is the failure: escalating the novel logic bug to a 2.6B local model cost **1.6 kJ over 194 s and did not close it**. That is a capability result, not a harness one. ## Pay once, free forever — measured The cascade's headline claim needs a close expensive enough to be worth caching. The local tiers cannot produce one on this suite, so this run enables L4 (the `claude` CLI) and takes two passes over `tasks/mix`. ``` pass task tier wall $ cloud J local J 1 const L1 205 ms $0.0000 0 21.0~ 1 greeting L1 822 ms $0.0000 0 0.0~ 1 logic L4 24 669 ms $0.3680 38 823 594.3 1 rename L1 552 ms $0.0000 0 0.0~ 1 typo L1 832 ms $0.0000 0 13.0~ 2 const L0 613 ms $0.0000 0 0.0~ 2 greeting L0 897 ms $0.0000 0 7.9~ 2 logic L0 582 ms $0.0000 0 0.0~ 2 rename L0 814 ms $0.0000 0 12.2~ 2 typo L0 733 ms $0.0000 0 15.4~ cascade 10/10 100.0% $0.0368/resolved 3 882 cloud J/resolved ``` The novel logic bug cost **$0.3680, 24.7 s and 38.8 kJ of (token-estimated) cloud energy** the first time. The second time it closed at **L0 for $0, no cloud, in 582 ms** — **42x faster**, with the entire cost eliminated. One cloud call amortises across ten resolutions to $0.0368 each. This is the first time this result has been produced with a working meter. Note what it is: a demonstration that the cache eliminates the cost of a repeated request, on a suite whose repetition rate is 100% by construction. It is not a claim about how often real work repeats, which this benchmark does not measure. ## The frontier, both solvers, same suite and passes `claude` (Claude Code 2.1.168) run over the identical suite and two passes: ``` solver resolved resol.% $/resolved cloud J/res local J/res wall/task cascade 10/10 100.0% $0.0368 3 882 66.4 0.2-24.7 s claude 10/10 100.0% $0.2809 35 830 220.6 13-29 s ``` **Identical resolution. 7.6x cheaper, 9.2x less cloud energy, 3.3x less local energy.** The mechanism is visible per pass rather than inferred: `claude` pays full price on the second pass exactly as on the first ($0.2688-$0.2791 against $0.2525-$0.3534), because a stateless cloud agent has nothing to amortise. The cascade's entire second pass closed at L0 for $0. Latency separates them further on the tiers that close locally: L0/L1 closes run 0.2-0.9 s against 13-29 s for a cloud round trip, though the cascade's own L4 escalation costs 24.7 s, which is the same round trip plus local overhead. **One prior belief corrected.** It was recorded here that a cloud agent spends "~0 local joules" because its compute happens in the vendor's datacenter. With a working meter it spends **220.6 J per resolved task** locally — the CLI process, orchestration and file I/O across a 13-29 s run are not free. The old figure was an artefact of the broken meter, not a property of cloud agents. ### What this multiple is and is not It is a measured result for **this suite**, which the section above shows is 80% L1-shaped against 2-3% in real commit history, and which repeats every task exactly once. Both properties favour the cascade, by construction. A workload with less repetition and more multi-file work would move the multiple toward 1, and nothing here bounds how far. ## The `real` suite: what the local tiers do on representative work `mix` over-represents L1-shaped work by 25-40x, so `tasks/real` was built to the measured distribution instead: six tasks, three whose fix spans two source files, two single-file rewrites of real logic, one small single-file fix. Ordinary bugs — a sliding window that never prunes, a parser folding operators left to right, a config loader silently accepting misspelled keys, interval merging that assumes sorted input, a retry loop that retries unretryable failures, a page count that drops the last partial page. The suite is sound: `reference` 6/6, `noop` 0/6. ``` solver resolved resol.% reference 6/6 100.0% noop 0/6 0.0% cascade 0/6 0.0% (local tiers only, no L4) ``` **The local tiers close none of it.** Against 4/5 on `mix`. None of these bugs is a substitution, a cached repeat, or a paraphrase of something already solved, so L0, L1 and L2 have nothing to offer and every task falls through. With L4 disabled they simply fail, in under a second each. This is the result the composition analysis predicted, and it bounds the cascade's local claim precisely: **the cheap tiers address a real but narrow class of work, and that class is not what fills a commit log.** The 7.6x advantage measured on `mix` comes from that suite's composition, not from a general property. The 0/6 was checked against my own mistake before being believed. The first version of these tasks had no `target` field, so the cascade ran with an empty document path and its single-file tiers had nothing to act on — a handicap I had introduced, not a property of the cascade. With `target` set on the four tasks whose fix really is single-file (the other two span two files, which is the point of including them), the result is unchanged: **still 0/6**. These bugs need reasoning — prune the list, reorder precedence, reject unknown keys, sort before merging, tell permanent failures from transient, round the division up — and none is a substitution, a cached repeat, or a paraphrase. ### Amortisation: single-file first, then multi-file **Before the fix.** Two passes, L4 enabled. Every single-file close came back at L0 for free on the repeat; neither multi-file close did — both paid the cloud in full again. The cascade resolved 11/12 at **$0.3176** per resolved task, no cheaper than raw `claude`. The cause was not the open standard, whose `Edit` has always carried its own path. The L4 capture read only the target file, so a fix spanning two files was seen as no change and never recorded; target-less requests were keyed on an empty path; and replay ignored `edit.path` altogether. Fixed in agent-joule `cfe83c4`. **After the fix.** The same two-pass run: ``` pass task shape tier $ 1 config-defaults MULTI-FILE L4 $0.5943 1 interval-merge single-file L4 $0.3927 1 pagination single-file L4 $0.4029 1 precedence single-file L4 $0.4417 1 rate-window MULTI-FILE L4 $0.3901 1 retry-backoff single-file L4 $0.4453 2 config-defaults MULTI-FILE L0 $0.0000 2 interval-merge single-file L0 $0.0000 2 pagination single-file L0 $0.0000 2 precedence single-file L0 $0.0000 2 rate-window MULTI-FILE L0 $0.0000 2 retry-backoff single-file L0 $0.0000 cascade 12/12 100.0% $0.2223/resolved ``` **All six repeats closed at L0, both multi-file changes included.** The cascade now pays exactly once per distinct task, whatever the shape of the change. Stated precisely, because the comparison is easy to overstate. With every task repeated once and every repeat cached, the cascade makes half the cloud calls a stateless agent does — so at equal per-call prices it costs exactly half, by construction. Against the `claude` baseline measured earlier ($0.2809) the gap is only 21%, because this run's per-call prices were higher ($0.39-$0.59 against $0.25-$0.35 then); per-call cost moves by more than 2x between runs, and those two figures come from different runs. The structural claim is the robust one. **Timing and local energy from this run are not valid.** Load average was 50-87 throughout, from unrelated work on the host. The clearest symptom is `rate-window` pass one reporting 90,155 J of local energy for a 25 s run — about 3.5 kW. Tier outcomes and dollar costs do not depend on CPU contention and are unaffected; everything else in this run should be ignored. ## What is not measured here **The agent loop.** `joule-coded agent` exists and joule-bench has an `agent` solver, but the on-device runtime currently generates at **1.73 tokens/sec** on this model — roughly twenty times slower than the hardware should manage. An agent-loop row today would measure the runtime, not the harness, so none is published. Enabling the loader's Metal decode driver (`jouleclaw-runtime/fabric`) was measured and is **worse**: 1.18 tok/s and 48.1 J/token against 1.73 tok/s and 33.8 J/token on the reference path. **Anything beyond nine tasks.** `mix` is 5, `smoke` 3, `cascade` 1. These suites show the machinery works; they are far too small to support a comparative claim, and as measured above their composition is not representative either. **Any neutral third-party harness.** We have never been scored on one. --- ## Superseded: the figures published before 2026-09-10 They reported `cascade 10/10 100%` and local energy `~0` on this suite. Three defects, each fatal to a different column: 1. **Every energy reading measured the meter.** One `read()` on the Apple model-based counter takes ~840 ms and reports the energy of its own sampling window; the meter bracketed each task between two of them. A 0 ms task read as 34 J. The counter is now sampled on a background thread. 2. **Idle draw was calibrated at about nine times reality.** The calibration slept 300 ms between two of those 840 ms reads, then divided an energy delta covering ~2 s by 0.3 s — reporting **352.2 W** idle for a Mac. It is **38 W**. That inflated figure was subtracted from every task, which is the real reason the old local-energy column read ~0. 3. **Resolution was under-reported on every Python task.** The baseline run left a `__pycache__` and the graded run imported the stale bytecode, so a correct fix still failed. `reference` — the ceiling — scored 4/5. Nothing about the harness's design was wrong; it could simply not measure. Both the energy and the resolution columns were unusable, in opposite directions.