# joule-bench

A coding-agent benchmark that scores **resolution rate and energy/cost together**,
producing a 2D Pareto frontier. Resolution is graded by an **executable test
oracle** — the task's tests fail before the solver runs and must pass after — so
the headline number is *task success*, not text similarity.

This is the harness behind the claim that Joule Code is SOTA "in the age of
agents": not by out-accuracy-ing frontier models on raw resolution, but by
resolving competitively at far lower joules and dollars, with the energy
attested per op.

## What it does

For each `(task, solver)`:

1. Seed a temp workdir with the task's pristine **buggy** tree (`before/`).
2. Confirm the bug reproduces (baseline tests **fail**) — else the datapoint is
   flagged invalid.
3. Bracket **energy** across the solver run.
4. Restore pristine **test files** (anti-gaming: a solver cannot pass by editing
   the test).
5. **Grade**: resolved iff the tests now pass.

Then aggregate per solver into frontier points: resolution %, joules per
resolved task (total energy ÷ resolved, matching the EoC `joules_per_correct`
metric), total joules, median wall time.

## Solvers

- `reference` — applies the task's known-good fix (`reference/`). The ceiling.
- `noop` — touches nothing. The floor (proves the oracle rejects unfixed code).
- `claude` — drives the `claude` CLI headlessly.
- `gemini` — drives the `gemini` CLI headlessly.

The agent-joule cascade solver (driving the headless `joule-coded` worker over
its control socket) and a SWE-bench Verified suite are the next increments.

## Usage

```sh
cargo run -- run --suite smoke --solvers reference,noop --out results.json
cargo run -- report --results results.json

# Test-time compute: best-of-N rollouts per task → the metered resolution × energy × $
# frontier (more rollouts resolve more, at a measured cost — the trade nobody else meters).
cargo run -- run --suite mix --solvers claude --rollouts 3 --out ttc.json
```

## Energy

Energy is captured through an `EnergyMeter` trait, selected with `--energy`:

- `measured` — jouleclaw host counter (Apple IOReport ModelBased without sudo,
  Linux RAPL / NVML HwShunt), the same path agent-joule meters with. Idle
  whole-SoC power is calibrated once and subtracted (`idle × wall`) to isolate
  the solver's *marginal* energy.
- `estimator` — wall-clock × active-watts, **always labeled `Estimator`** (a
  coefficient, never a measurement). Override with `JOULE_BENCH_ACTIVE_WATTS`.
- `auto` (default) — measured if a host counter reads, else estimator.

**Measurement floor (honesty).** A whole-SoC counter cannot resolve work shorter
than its minimum window (~10 ms for IOReport). Any run below that window is
flagged `~` and reported as energy-unreliable: the *resolution* result stays
valid, but the energy figure is noise-dominated and not comparable. Trivial
solvers (`reference`, `noop`) finish in microseconds and are always flagged —
real energy signal comes from multi-second agent runs (the `claude`/cascade
solvers), where the solver's work dominates the window.

## Tasks

A task is a directory under `tasks/<suite>/<name>/`:

```
task.toml      # name, prompt, test_cmd, test_files (restored before grading)
before/        # pristine buggy tree (source + tests), seeded into the workdir
reference/     # known-good fix, applied by the `reference` solver
```