- Rust 83.2%
- Python 16.8%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
740b6b21d2
Re-ran the two-pass L4 measurement on `real` after agent-joule cfe83c4. Every repeat closed at L0 for $0, including both multi-file changes that previously paid the cloud in full on every pass. The cascade now pays exactly once per distinct task regardless of how many files the fix touches, and $/resolved falls from $0.3176 to $0.2223. The comparison is stated narrowly on purpose. With every task repeated once and every repeat cached, the cascade makes half the cloud calls a stateless agent does, so at equal per-call prices it costs exactly half — by construction, not as a discovery. Against the earlier claude baseline the gap is only 21%, because this run's per-call prices were higher and the two figures come from different runs. The structural claim is the robust one; the percentage is not. Timing and local energy from this run are explicitly marked invalid. The host sat at load average 50-87 from unrelated work, and one row reports 90,155 J of local energy for a 25 s run — about 3.5 kW. Tier outcomes and dollar costs are not affected by CPU contention and are reported; nothing else from this run should be read. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
| src | ||
| swebench | ||
| tasks | ||
| .gitignore | ||
| Cargo.lock | ||
| Cargo.toml | ||
| claude.joule-consensus-val-r1.json | ||
| claude.joule-consensus-val-r2.json | ||
| claude.joule-consensus-val-r3.json | ||
| README.md | ||
| RESULTS.md | ||
| results_mix.json | ||
| results_real.json | ||
joule-bench
A coding-agent benchmark that scores resolution rate and energy/cost together, producing a 2D Pareto frontier. Resolution is graded by an executable test oracle — the task's tests fail before the solver runs and must pass after — so the headline number is task success, not text similarity.
This is the harness behind the claim that Joule Code is SOTA "in the age of agents": not by out-accuracy-ing frontier models on raw resolution, but by resolving competitively at far lower joules and dollars, with the energy attested per op.
What it does
For each (task, solver):
- Seed a temp workdir with the task's pristine buggy tree (
before/). - Confirm the bug reproduces (baseline tests fail) — else the datapoint is flagged invalid.
- Bracket energy across the solver run.
- Restore pristine test files (anti-gaming: a solver cannot pass by editing the test).
- Grade: resolved iff the tests now pass.
Then aggregate per solver into frontier points: resolution %, joules per
resolved task (total energy ÷ resolved, matching the EoC joules_per_correct
metric), total joules, median wall time.
Solvers
reference— applies the task's known-good fix (reference/). The ceiling.noop— touches nothing. The floor (proves the oracle rejects unfixed code).claude— drives theclaudeCLI headlessly.gemini— drives thegeminiCLI headlessly.
The agent-joule cascade solver (driving the headless joule-coded worker over
its control socket) and a SWE-bench Verified suite are the next increments.
Usage
cargo run -- run --suite smoke --solvers reference,noop --out results.json
cargo run -- report --results results.json
# Test-time compute: best-of-N rollouts per task → the metered resolution × energy × $
# frontier (more rollouts resolve more, at a measured cost — the trade nobody else meters).
cargo run -- run --suite mix --solvers claude --rollouts 3 --out ttc.json
Energy
Energy is captured through an EnergyMeter trait, selected with --energy:
measured— jouleclaw host counter (Apple IOReport ModelBased without sudo, Linux RAPL / NVML HwShunt), the same path agent-joule meters with. Idle whole-SoC power is calibrated once and subtracted (idle × wall) to isolate the solver's marginal energy.estimator— wall-clock × active-watts, always labeledEstimator(a coefficient, never a measurement). Override withJOULE_BENCH_ACTIVE_WATTS.auto(default) — measured if a host counter reads, else estimator.
Measurement floor (honesty). A whole-SoC counter cannot resolve work shorter
than its minimum window (~10 ms for IOReport). Any run below that window is
flagged ~ and reported as energy-unreliable: the resolution result stays
valid, but the energy figure is noise-dominated and not comparable. Trivial
solvers (reference, noop) finish in microseconds and are always flagged —
real energy signal comes from multi-second agent runs (the claude/cascade
solvers), where the solver's work dominates the window.
Tasks
A task is a directory under tasks/<suite>/<name>/:
task.toml # name, prompt, test_cmd, test_files (restored before grading)
before/ # pristine buggy tree (source + tests), seeded into the workdir
reference/ # known-good fix, applied by the `reference` solver