- Rust 83.2%
- Python 16.8%
367ce74caa
run_swebench.py runs N independent rollouts per instance and reports the best-of-k frontier: resolution within the first k rollouts vs cumulative dollars. First best-of-3 sample stayed flat at 2/3 while cost grew 3x — the failure was deterministic, so even oracle selection (the upper bound) cannot lift it. That is the value of metering: it shows when test-time compute is not paying off. The literature's ~7pp lift needs stronger per-rollout agents + summarize-then-vote selection, not naive best-of-N; documented as the next builds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| src | ||
| swebench | ||
| tasks | ||
| .gitignore | ||
| Cargo.lock | ||
| Cargo.toml | ||
| README.md | ||
| RESULTS.md | ||
| results_mix.json | ||
| results_real.json | ||
joule-bench
A coding-agent benchmark that scores resolution rate and energy/cost together, producing a 2D Pareto frontier. Resolution is graded by an executable test oracle — the task's tests fail before the solver runs and must pass after — so the headline number is task success, not text similarity.
This is the harness behind the claim that Joule Code is SOTA "in the age of agents": not by out-accuracy-ing frontier models on raw resolution, but by resolving competitively at far lower joules and dollars, with the energy attested per op.
What it does
For each (task, solver):
- Seed a temp workdir with the task's pristine buggy tree (
before/). - Confirm the bug reproduces (baseline tests fail) — else the datapoint is flagged invalid.
- Bracket energy across the solver run.
- Restore pristine test files (anti-gaming: a solver cannot pass by editing the test).
- Grade: resolved iff the tests now pass.
Then aggregate per solver into frontier points: resolution %, joules per
resolved task (total energy ÷ resolved, matching the EoC joules_per_correct
metric), total joules, median wall time.
Solvers
reference— applies the task's known-good fix (reference/). The ceiling.noop— touches nothing. The floor (proves the oracle rejects unfixed code).claude— drives theclaudeCLI headlessly.gemini— drives thegeminiCLI headlessly.
The agent-joule cascade solver (driving the headless joule-coded worker over
its control socket) and a SWE-bench Verified suite are the next increments.
Usage
cargo run -- run --suite smoke --solvers reference,noop --out results.json
cargo run -- report --results results.json
# Test-time compute: best-of-N rollouts per task → the metered resolution × energy × $
# frontier (more rollouts resolve more, at a measured cost — the trade nobody else meters).
cargo run -- run --suite mix --solvers claude --rollouts 3 --out ttc.json
Energy
Energy is captured through an EnergyMeter trait, selected with --energy:
measured— jouleclaw host counter (Apple IOReport ModelBased without sudo, Linux RAPL / NVML HwShunt), the same path agent-joule meters with. Idle whole-SoC power is calibrated once and subtracted (idle × wall) to isolate the solver's marginal energy.estimator— wall-clock × active-watts, always labeledEstimator(a coefficient, never a measurement). Override withJOULE_BENCH_ACTIVE_WATTS.auto(default) — measured if a host counter reads, else estimator.
Measurement floor (honesty). A whole-SoC counter cannot resolve work shorter
than its minimum window (~10 ms for IOReport). Any run below that window is
flagged ~ and reported as energy-unreliable: the resolution result stays
valid, but the energy figure is noise-dominated and not comparable. Trivial
solvers (reference, noop) finish in microseconds and are always flagged —
real energy signal comes from multi-second agent runs (the claude/cascade
solvers), where the solver's work dominates the window.
Tasks
A task is a directory under tasks/<suite>/<name>/:
task.toml # name, prompt, test_cmd, test_files (restored before grading)
before/ # pristine buggy tree (source + tests), seeded into the workdir
reference/ # known-good fix, applied by the `reference` solver