joule-bench/RESULTS.md

# joule-bench results

Re-measured 2026-09-10 on Apple Silicon, after fixing three defects that made
every earlier figure in this file wrong. The old numbers and what was wrong with
them are kept at the bottom rather than deleted.

Resolution is graded by an executable test oracle: the task's tests must fail
before the solver runs and pass after, with pristine test files restored first so
a solver cannot pass by rewriting the test. Local energy is host-counter
measured, idle-subtracted (idle calibrated per run at **36.9–38.4 W**).

## The `mix` suite (and what it is not)

Suite `tasks/mix`: four mechanical edits (rename, typo, constant, string) plus
one novel logic bug. One pass, L4 disabled, so this is **purely local**.

**This suite is not a workload model, and it used to claim to be.** It was called
a "realistic operation mix" on the grounds that mechanical edits dominate real
work. Measured against actual history, they do not. Classifying 1,679
source-touching commits across three real repositories:

```
                            joulesperbit   open-standards   agent-joule
multi-file                       53.4%           66.0%          62.9%
single-file, large               37.8%           30.3%          27.8%
single-file, small (<=20 lines)   8.8%            3.7%           9.4%
single-file, <=10 lines           3.3%            2.3%             —
strict one-token rename           0.0%            0.0%           0.0%
```

Not one commit in 1,679 was a single-file consistent one-token rename. On the
generous reading — any single-file change under ten lines — real work is
**2.3-3.3%** of that shape. `mix` is **80%**. So the suite over-represents the
cascade's cheapest tier by roughly **25-40x**, and a resolution figure from it
must not be read as a workload result.

What the measurement cannot see is repetition. Commit history records what
changed, never how often someone asked for something similar, and L0 (cache) and
L2 (retrieval) exist for exactly that. So this understates the local tiers'
real-world reach by an unknown amount — but the correction is unmeasured, and
pointing at it is not a defence of the 80%.

What the suite does establish is narrower and still worth having: **when work is
L1-shaped, it closes deterministically, sub-second, at energy below the
measurement floor, and configuring a model does not change that.**

```
solver       resolved  resol.%   note
reference       5/5     100.0%   applies each task's known-good patch — the ceiling
cascade         4/5      80.0%   four closed at L1; the logic bug did not close
noop            0/5       0.0%   touches nothing — the floor
```

The ceiling and floor are both where they should be, which is the first time that
has been true. Per task, with no model configured:

```
task       tier   wall     resolved
const      L1     1457 ms  yes
greeting   L1      291 ms  yes
rename     L1      346 ms  yes
typo       L1      824 ms  yes
logic       —      415 ms  no
```

## Configuring a local model no longer taxes work that does not need it

The same suite with a 1.6 GB LFM2-2.6B-Q4 wired in as L3:

```
task       tier   wall     energy      resolved
const      L1      518 ms   0 J~        yes
greeting   L1      557 ms   0 J~        yes
rename     L1      525 ms  17.9 J~      yes
typo       L1      421 ms   0 J~        yes
logic       —   193991 ms  1616 J       no   (L3 attempted, did not close)
```

The four L1 closes cost the same with a model configured as without one. That is
the point of the cascade and it was **not** true a week ago: before the lazy-load
fix these same four tasks cost **52–92 J and 4.7–10.5 s each**, all of it spent
loading a model none of them touched. Two causes — eager construction, and an
L2-index embedding computed after *every* close, including deterministic ones,
which forced the load anyway. Indexing now happens only when the model is already
resident.

`~` marks runs shorter than the counter's minimum sampling window. Their
resolution is valid; their energy is not comparable and the harness says so
rather than printing a fabricated number.

The one energy figure here that is properly measurable is the failure: escalating
the novel logic bug to a 2.6B local model cost **1.6 kJ over 194 s and did not
close it**. That is a capability result, not a harness one.

## Pay once, free forever — measured

The cascade's headline claim needs a close expensive enough to be worth caching.
The local tiers cannot produce one on this suite, so this run enables L4 (the
`claude` CLI) and takes two passes over `tasks/mix`.

```
pass task       tier   wall       $         cloud J    local J
1    const      L1        205 ms  $0.0000        0      21.0~
1    greeting   L1        822 ms  $0.0000        0       0.0~
1    logic      L4     24 669 ms  $0.3680   38 823     594.3
1    rename     L1        552 ms  $0.0000        0       0.0~
1    typo       L1        832 ms  $0.0000        0      13.0~
2    const      L0        613 ms  $0.0000        0       0.0~
2    greeting   L0        897 ms  $0.0000        0       7.9~
2    logic      L0        582 ms  $0.0000        0       0.0~
2    rename     L0        814 ms  $0.0000        0      12.2~
2    typo       L0        733 ms  $0.0000        0      15.4~

cascade   10/10   100.0%   $0.0368/resolved   3 882 cloud J/resolved
```

The novel logic bug cost **$0.3680, 24.7 s and 38.8 kJ of (token-estimated) cloud
energy** the first time. The second time it closed at **L0 for $0, no cloud, in
582 ms** — **42x faster**, with the entire cost eliminated. One cloud call
amortises across ten resolutions to $0.0368 each.

This is the first time this result has been produced with a working meter. Note
what it is: a demonstration that the cache eliminates the cost of a repeated
request, on a suite whose repetition rate is 100% by construction. It is not a
claim about how often real work repeats, which this benchmark does not measure.

## The frontier, both solvers, same suite and passes

`claude` (Claude Code 2.1.168) run over the identical suite and two passes:

```
solver     resolved  resol.%   $/resolved   cloud J/res   local J/res   wall/task
cascade      10/10    100.0%      $0.0368         3 882          66.4   0.2-24.7 s
claude       10/10    100.0%      $0.2809        35 830         220.6    13-29 s
```

**Identical resolution. 7.6x cheaper, 9.2x less cloud energy, 3.3x less local
energy.** The mechanism is visible per pass rather than inferred: `claude` pays
full price on the second pass exactly as on the first ($0.2688-$0.2791 against
$0.2525-$0.3534), because a stateless cloud agent has nothing to amortise. The
cascade's entire second pass closed at L0 for $0.

Latency separates them further on the tiers that close locally: L0/L1 closes run
0.2-0.9 s against 13-29 s for a cloud round trip, though the cascade's own L4
escalation costs 24.7 s, which is the same round trip plus local overhead.

**One prior belief corrected.** It was recorded here that a cloud agent spends
"~0 local joules" because its compute happens in the vendor's datacenter. With a
working meter it spends **220.6 J per resolved task** locally — the CLI process,
orchestration and file I/O across a 13-29 s run are not free. The old figure was
an artefact of the broken meter, not a property of cloud agents.

### What this multiple is and is not

It is a measured result for **this suite**, which the section above shows is
80% L1-shaped against 2-3% in real commit history, and which repeats every task
exactly once. Both properties favour the cascade, by construction. A workload
with less repetition and more multi-file work would move the multiple toward 1,
and nothing here bounds how far.

## The `real` suite: what the local tiers do on representative work

`mix` over-represents L1-shaped work by 25-40x, so `tasks/real` was built to the
measured distribution instead: six tasks, three whose fix spans two source files,
two single-file rewrites of real logic, one small single-file fix. Ordinary bugs
— a sliding window that never prunes, a parser folding operators left to right, a
config loader silently accepting misspelled keys, interval merging that assumes
sorted input, a retry loop that retries unretryable failures, a page count that
drops the last partial page.

The suite is sound: `reference` 6/6, `noop` 0/6.

```
solver     resolved  resol.%
reference     6/6     100.0%
noop          0/6       0.0%
cascade       0/6       0.0%   (local tiers only, no L4)
```

**The local tiers close none of it.** Against 4/5 on `mix`. None of these bugs is
a substitution, a cached repeat, or a paraphrase of something already solved, so
L0, L1 and L2 have nothing to offer and every task falls through. With L4
disabled they simply fail, in under a second each.

This is the result the composition analysis predicted, and it bounds the
cascade's local claim precisely: **the cheap tiers address a real but narrow class
of work, and that class is not what fills a commit log.** The 7.6x advantage
measured on `mix` comes from that suite's composition, not from a general
property.

The 0/6 was checked against my own mistake before being believed. The first
version of these tasks had no `target` field, so the cascade ran with an empty
document path and its single-file tiers had nothing to act on — a handicap I had
introduced, not a property of the cascade. With `target` set on the four tasks
whose fix really is single-file (the other two span two files, which is the point
of including them), the result is unchanged: **still 0/6**. These bugs need
reasoning — prune the list, reorder precedence, reject unknown keys, sort before
merging, tell permanent failures from transient, round the division up — and none
is a substitution, a cached repeat, or a paraphrase.

### Amortisation: single-file first, then multi-file

**Before the fix.** Two passes, L4 enabled. Every single-file close came back at
L0 for free on the repeat; neither multi-file close did — both paid the cloud in
full again. The cascade resolved 11/12 at **$0.3176** per resolved task, no
cheaper than raw `claude`.

The cause was not the open standard, whose `Edit` has always carried its own path.
The L4 capture read only the target file, so a fix spanning two files was seen as
no change and never recorded; target-less requests were keyed on an empty path;
and replay ignored `edit.path` altogether. Fixed in agent-joule `cfe83c4`.

**After the fix.** The same two-pass run:

```
pass task              shape         tier    $
1    config-defaults   MULTI-FILE    L4   $0.5943
1    interval-merge    single-file   L4   $0.3927
1    pagination        single-file   L4   $0.4029
1    precedence        single-file   L4   $0.4417
1    rate-window       MULTI-FILE    L4   $0.3901
1    retry-backoff     single-file   L4   $0.4453
2    config-defaults   MULTI-FILE    L0   $0.0000
2    interval-merge    single-file   L0   $0.0000
2    pagination        single-file   L0   $0.0000
2    precedence        single-file   L0   $0.0000
2    rate-window       MULTI-FILE    L0   $0.0000
2    retry-backoff     single-file   L0   $0.0000

cascade  12/12  100.0%  $0.2223/resolved
```

**All six repeats closed at L0, both multi-file changes included.** The cascade
now pays exactly once per distinct task, whatever the shape of the change.

Stated precisely, because the comparison is easy to overstate. With every task
repeated once and every repeat cached, the cascade makes half the cloud calls a
stateless agent does — so at equal per-call prices it costs exactly half, by
construction. Against the `claude` baseline measured earlier ($0.2809) the gap is
only 21%, because this run's per-call prices were higher ($0.39-$0.59 against
$0.25-$0.35 then); per-call cost moves by more than 2x between runs, and those two
figures come from different runs. The structural claim is the robust one.

**Timing and local energy from this run are not valid.** Load average was 50-87
throughout, from unrelated work on the host. The clearest symptom is `rate-window`
pass one reporting 90,155 J of local energy for a 25 s run — about 3.5 kW. Tier
outcomes and dollar costs do not depend on CPU contention and are unaffected;
everything else in this run should be ignored.

## What is not measured here

**The agent loop.** `joule-coded agent` exists and joule-bench has an `agent`
solver, but the on-device runtime currently generates at **1.73 tokens/sec** on
this model — roughly twenty times slower than the hardware should manage. An
agent-loop row today would measure the runtime, not the harness, so none is
published. Enabling the loader's Metal decode driver (`jouleclaw-runtime/fabric`)
was measured and is **worse**: 1.18 tok/s and 48.1 J/token against 1.73 tok/s and
33.8 J/token on the reference path.

**Anything beyond nine tasks.** `mix` is 5, `smoke` 3, `cascade` 1. These suites
show the machinery works; they are far too small to support a comparative claim,
and as measured above their composition is not representative either.

**Any neutral third-party harness.** We have never been scored on one.

---

## Superseded: the figures published before 2026-09-10

They reported `cascade 10/10 100%` and local energy `~0` on this suite. Three
defects, each fatal to a different column:

1. **Every energy reading measured the meter.** One `read()` on the Apple
   model-based counter takes ~840 ms and reports the energy of its own sampling
   window; the meter bracketed each task between two of them. A 0 ms task read as
   34 J. The counter is now sampled on a background thread.
2. **Idle draw was calibrated at about nine times reality.** The calibration slept
   300 ms between two of those 840 ms reads, then divided an energy delta covering
   ~2 s by 0.3 s — reporting **352.2 W** idle for a Mac. It is **38 W**. That
   inflated figure was subtracted from every task, which is the real reason the
   old local-energy column read ~0.
3. **Resolution was under-reported on every Python task.** The baseline run left a
   `__pycache__` and the graded run imported the stale bytecode, so a correct fix
   still failed. `reference` — the ceiling — scored 4/5.

Nothing about the harness's design was wrong; it could simply not measure. Both
the energy and the resolution columns were unusable, in opposite directions.