712252b2a7
Run N independent rollouts per (solver, task, pass) and report the best-of-k frontier: for each k=1..N, the fraction of tasks resolved within the first k rollouts paired with the cumulative dollars + cloud/local energy those k rollouts spent. This is the resolution × energy × $ curve of test-time compute — the trade no other harness meters. The recent literature shows best-of-N + rollout selection lifts frontier agents ~7pp on SWE-bench Verified; this measures what that lift costs in joules and dollars. TaskRun gains a rollout index; test_time_frontier() aggregates best-of-k; render shows the curve. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
78 lines
3.3 KiB
Markdown
78 lines
3.3 KiB
Markdown
# joule-bench
|
||
|
||
A coding-agent benchmark that scores **resolution rate and energy/cost together**,
|
||
producing a 2D Pareto frontier. Resolution is graded by an **executable test
|
||
oracle** — the task's tests fail before the solver runs and must pass after — so
|
||
the headline number is *task success*, not text similarity.
|
||
|
||
This is the harness behind the claim that Joule Code is SOTA "in the age of
|
||
agents": not by out-accuracy-ing frontier models on raw resolution, but by
|
||
resolving competitively at far lower joules and dollars, with the energy
|
||
attested per op.
|
||
|
||
## What it does
|
||
|
||
For each `(task, solver)`:
|
||
|
||
1. Seed a temp workdir with the task's pristine **buggy** tree (`before/`).
|
||
2. Confirm the bug reproduces (baseline tests **fail**) — else the datapoint is
|
||
flagged invalid.
|
||
3. Bracket **energy** across the solver run.
|
||
4. Restore pristine **test files** (anti-gaming: a solver cannot pass by editing
|
||
the test).
|
||
5. **Grade**: resolved iff the tests now pass.
|
||
|
||
Then aggregate per solver into frontier points: resolution %, joules per
|
||
resolved task (total energy ÷ resolved, matching the EoC `joules_per_correct`
|
||
metric), total joules, median wall time.
|
||
|
||
## Solvers
|
||
|
||
- `reference` — applies the task's known-good fix (`reference/`). The ceiling.
|
||
- `noop` — touches nothing. The floor (proves the oracle rejects unfixed code).
|
||
- `claude` — drives the `claude` CLI headlessly.
|
||
- `gemini` — drives the `gemini` CLI headlessly.
|
||
|
||
The agent-joule cascade solver (driving the headless `joule-coded` worker over
|
||
its control socket) and a SWE-bench Verified suite are the next increments.
|
||
|
||
## Usage
|
||
|
||
```sh
|
||
cargo run -- run --suite smoke --solvers reference,noop --out results.json
|
||
cargo run -- report --results results.json
|
||
|
||
# Test-time compute: best-of-N rollouts per task → the metered resolution × energy × $
|
||
# frontier (more rollouts resolve more, at a measured cost — the trade nobody else meters).
|
||
cargo run -- run --suite mix --solvers claude --rollouts 3 --out ttc.json
|
||
```
|
||
|
||
## Energy
|
||
|
||
Energy is captured through an `EnergyMeter` trait, selected with `--energy`:
|
||
|
||
- `measured` — jouleclaw host counter (Apple IOReport ModelBased without sudo,
|
||
Linux RAPL / NVML HwShunt), the same path agent-joule meters with. Idle
|
||
whole-SoC power is calibrated once and subtracted (`idle × wall`) to isolate
|
||
the solver's *marginal* energy.
|
||
- `estimator` — wall-clock × active-watts, **always labeled `Estimator`** (a
|
||
coefficient, never a measurement). Override with `JOULE_BENCH_ACTIVE_WATTS`.
|
||
- `auto` (default) — measured if a host counter reads, else estimator.
|
||
|
||
**Measurement floor (honesty).** A whole-SoC counter cannot resolve work shorter
|
||
than its minimum window (~10 ms for IOReport). Any run below that window is
|
||
flagged `~` and reported as energy-unreliable: the *resolution* result stays
|
||
valid, but the energy figure is noise-dominated and not comparable. Trivial
|
||
solvers (`reference`, `noop`) finish in microseconds and are always flagged —
|
||
real energy signal comes from multi-second agent runs (the `claude`/cascade
|
||
solvers), where the solver's work dominates the window.
|
||
|
||
## Tasks
|
||
|
||
A task is a directory under `tasks/<suite>/<name>/`:
|
||
|
||
```
|
||
task.toml # name, prompt, test_cmd, test_files (restored before grading)
|
||
before/ # pristine buggy tree (source + tests), seeded into the workdir
|
||
reference/ # known-good fix, applied by the `reference` solver
|
||
```
|