joule-bench/README.md

joule-bench

A coding-agent benchmark that scores resolution rate and energy/cost together, producing a 2D Pareto frontier. Resolution is graded by an executable test oracle — the task's tests fail before the solver runs and must pass after — so the headline number is task success, not text similarity.

This is the harness behind the claim that Joule Code is SOTA "in the age of agents": not by out-accuracy-ing frontier models on raw resolution, but by resolving competitively at far lower joules and dollars, with the energy attested per op.

What it does

For each (task, solver):

1. Seed a temp workdir with the task's pristine buggy tree (before/).
2. Confirm the bug reproduces (baseline tests fail) — else the datapoint is flagged invalid.
3. Bracket energy across the solver run.
4. Restore pristine test files (anti-gaming: a solver cannot pass by editing the test).
5. Grade: resolved iff the tests now pass.

Then aggregate per solver into frontier points: resolution %, joules per resolved task (total energy ÷ resolved, matching the EoC joules_per_correct metric), total joules, median wall time.

Solvers

• reference — applies the task's known-good fix (reference/). The ceiling.
• noop — touches nothing. The floor (proves the oracle rejects unfixed code).
• claude — drives the claude CLI headlessly.
• gemini — drives the gemini CLI headlessly.

The agent-joule cascade solver (driving the headless joule-coded worker over its control socket) and a SWE-bench Verified suite are the next increments.

Usage

cargo run -- run --suite smoke --solvers reference,noop --out results.json
cargo run -- report --results results.json

# Test-time compute: best-of-N rollouts per task → the metered resolution × energy × $
# frontier (more rollouts resolve more, at a measured cost — the trade nobody else meters).
cargo run -- run --suite mix --solvers claude --rollouts 3 --out ttc.json

Energy

Energy is captured through an EnergyMeter trait, selected with --energy:

• measured — jouleclaw host counter (Apple IOReport ModelBased without sudo, Linux RAPL / NVML HwShunt), the same path agent-joule meters with. Idle whole-SoC power is calibrated once and subtracted (idle × wall) to isolate the solver's marginal energy.
• estimator — wall-clock × active-watts, always labeled Estimator (a coefficient, never a measurement). Override with JOULE_BENCH_ACTIVE_WATTS.
• auto (default) — measured if a host counter reads, else estimator.

Measurement floor (honesty). A whole-SoC counter cannot resolve work shorter than its minimum window (~10 ms for IOReport). Any run below that window is flagged ~ and reported as energy-unreliable: the resolution result stays valid, but the energy figure is noise-dominated and not comparable. Trivial solvers (reference, noop) finish in microseconds and are always flagged — real energy signal comes from multi-second agent runs (the claude/cascade solvers), where the solver's work dominates the window.

Tasks

A task is a directory under tasks/<suite>/<name>/:

task.toml      # name, prompt, test_cmd, test_files (restored before grading)
before/        # pristine buggy tree (source + tests), seeded into the workdir
reference/     # known-good fix, applied by the `reference` solver