joule-bench/swebench

SWE-bench Verified — external parity check

This runs a solver over a subset of SWE-bench Verified (500 human-validated GitHub issues) through the official swebench Docker evaluation harness — the externally-recognized coding-agent benchmark. It complements the energy/cost frontier in ../RESULTS.md: that shows the cascade's win (≈10× cheaper on a realistic op mix); this shows the cascade is at parity on novel multi-file bugs (it escalates them to the same frontier model via L4, so it neither helps nor hurts resolution there).

Setup (macOS, Apple Silicon)

open -a OrbStack                       # a Docker daemon (fast; emulates linux/amd64)
python3.11 -m venv ~/.swebench-venv    # swebench needs Python 3.10+
~/.swebench-venv/bin/pip install swebench datasets
~/.swebench-venv/bin/python swebench/run_swebench.py --solver claude \
    --instances psf__requests-5414 pylint-dev__pylint-7080

The SWE-bench prebuilt images are linux/amd64; on Apple Silicon they build/run under emulation (slower, occasionally flaky). 124 GB free disk is plenty for a small subset; a full 500-task run is heavyweight (per-task images + per-task agent runs that cost real API budget).

What it measures, honestly

A run produces resolution rate (the harness builds the task image, applies the solver's source patch + the hidden test patch, and runs FAIL_TO_PASS + PASS_TO_PASS) paired with the solver's dollar cost.

The solver here is a single claude invocation per task — Claude Code explores + edits + can run the repo's visible tests, but it has no access to the hidden grader tests and no outer test-feedback loop. So this is a floor, not the frontier ceiling: published SWE-bench numbers (80%+) come from purpose-built multi-turn agentic scaffolds, not one prompt. The number's absolute value is a property of the scaffold, not the model.

Sample (9 instances, real Docker eval, aarch64/OrbStack)

instance                     resolved   $ cost
psf__requests-1921           ✗ no       $0.299    (wrong fix; broke 24 tests)
psf__requests-5414           ✓ yes      $0.538
psf__requests-6028           (empty)    $0.260    (agent errored, no patch)
pylint-dev__pylint-7080      ✓ yes      $2.893
pallets__flask-5014          ✓ yes      $0.310
psf__requests-1766           ✗ no       $0.425
psf__requests-2931           ✓ yes      $1.327
pylint-dev__pylint-4970      ✗ no       $0.527
pylint-dev__pylint-7277      ✓ yes      $0.759
─────────────────────────────────────────────────────
RESOLVED 5/9 = 56%   total $7.34   $1.47/resolved

(Run as two batches via run_swebench.py; the second 5-instance batch resolved 3/5 at $1.12/resolved.) Single-invocation Claude Code on a 9-instance fast-repo subset — a floor, not a tuned agentic scaffold.

Metered test-time compute (--rollouts N)

--rollouts N runs N independent rollouts per instance and reports the best-of-k frontier paired with cumulative dollars — the resolution × $ curve of test-time compute, metered (no other harness measures the cost of the lift). Each k now reports two resolution numbers:

• oracle — resolved if any of the first k rollouts resolves. The ceiling of any selection method; not realizable (you'd need the hidden grader to know which rollout to ship).
• consensus — the no-leak BFT tournament vote (consensus_pick) actually selects one candidate per instance: the patch the most rollouts agree on (plurality over normalized diffs, ties → earliest rollout), never consulting the grader. This is the resolution a realizable selector achieves. The cost is identical to oracle — all k rollouts run either way — so the oracle−consensus gap is pure selection-quality signal, not a cost trade.

 k   oracle    consensus   cum $
 1     2/3        2/3       2.18
 2     2/3        2/3       4.19
 3     2/3        2/3       6.24

Honest reading: this sample's failure was deterministic, so even the oracle ceiling stays flat while cost grows 3× — metering lets you see test-time compute isn't paying off and stop. When rollouts do disagree, consensus is what closes the oracle gap without peeking at the grader: independent metered workers vote and ship the agreed patch (this maps directly onto agent-joule's lockstep-quic BFT fleet — the vote is real consensus, not a heuristic). Realizing the literature's ~7pp lift further needs stronger per-rollout agents (harness engineering — tools/middleware/memory, e.g. the new recall long-term-memory tool in the L3 loop). The selector and its metering are now built; the remaining lever is agent strength per rollout.

The harness itself is the deliverable

The harness runs end-to-end with zero errors and full per-test breakdown — joule-bench can now produce real SWE-bench Verified numbers. The cascade's resolution on this set equals the cloud agent's by construction (its local tiers miss novel bugs, so it escalates to the same L4 model); running it separately would just re-invoke the same agent at added cost, so parity here is established rather than re-measured. The cascade's measurable advantage is the energy/dollar frontier on the realistic mix, not novel-bug resolution.