367ce74caa
run_swebench.py runs N independent rollouts per instance and reports the best-of-k frontier: resolution within the first k rollouts vs cumulative dollars. First best-of-3 sample stayed flat at 2/3 while cost grew 3x — the failure was deterministic, so even oracle selection (the upper bound) cannot lift it. That is the value of metering: it shows when test-time compute is not paying off. The literature's ~7pp lift needs stronger per-rollout agents + summarize-then-vote selection, not naive best-of-N; documented as the next builds. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
4.4 KiB
SWE-bench Verified — external parity check
This runs a solver over a subset of SWE-bench Verified
(500 human-validated GitHub issues) through the official swebench Docker
evaluation harness — the externally-recognized coding-agent benchmark. It
complements the energy/cost frontier in ../RESULTS.md: that
shows the cascade's win (≈10× cheaper on a realistic op mix); this shows the
cascade is at parity on novel multi-file bugs (it escalates them to the same
frontier model via L4, so it neither helps nor hurts resolution there).
Setup (macOS, Apple Silicon)
open -a OrbStack # a Docker daemon (fast; emulates linux/amd64)
python3.11 -m venv ~/.swebench-venv # swebench needs Python 3.10+
~/.swebench-venv/bin/pip install swebench datasets
~/.swebench-venv/bin/python swebench/run_swebench.py --solver claude \
--instances psf__requests-5414 pylint-dev__pylint-7080
The SWE-bench prebuilt images are linux/amd64; on Apple Silicon they build/run
under emulation (slower, occasionally flaky). 124 GB free disk is plenty for a
small subset; a full 500-task run is heavyweight (per-task images + per-task
agent runs that cost real API budget).
What it measures, honestly
A run produces resolution rate (the harness builds the task image, applies the solver's source patch + the hidden test patch, and runs FAIL_TO_PASS + PASS_TO_PASS) paired with the solver's dollar cost.
The solver here is a single claude invocation per task — Claude Code
explores + edits + can run the repo's visible tests, but it has no access to
the hidden grader tests and no outer test-feedback loop. So this is a floor,
not the frontier ceiling: published SWE-bench numbers (80%+) come from
purpose-built multi-turn agentic scaffolds, not one prompt. The number's
absolute value is a property of the scaffold, not the model.
Sample (9 instances, real Docker eval, aarch64/OrbStack)
instance resolved $ cost
psf__requests-1921 ✗ no $0.299 (wrong fix; broke 24 tests)
psf__requests-5414 ✓ yes $0.538
psf__requests-6028 (empty) $0.260 (agent errored, no patch)
pylint-dev__pylint-7080 ✓ yes $2.893
pallets__flask-5014 ✓ yes $0.310
psf__requests-1766 ✗ no $0.425
psf__requests-2931 ✓ yes $1.327
pylint-dev__pylint-4970 ✗ no $0.527
pylint-dev__pylint-7277 ✓ yes $0.759
─────────────────────────────────────────────────────
RESOLVED 5/9 = 56% total $7.34 $1.47/resolved
(Run as two batches via run_swebench.py; the second 5-instance batch resolved
3/5 at $1.12/resolved.) Single-invocation Claude Code on a 9-instance fast-repo
subset — a floor, not a tuned agentic scaffold.
Metered test-time compute (--rollouts N)
--rollouts N runs N independent rollouts per instance and reports the best-of-k
frontier: resolution within the first k rollouts vs the cumulative dollars they
cost — the resolution × $ curve of test-time compute, metered (no other
harness measures the cost of the lift). First best-of-3 sample (3 requests
instances):
k resolved resol.% cum $
1 2/3 67% 2.18
2 2/3 67% 4.19
3 2/3 67% 6.24
Honest reading: this sample's failure was deterministic, so even oracle selection (resolved-if-any — the upper bound of any selection method) stays flat while cost grows 3×. That is the value of metering: you can see test-time compute isn't paying off and stop. Realizing the literature's ~7pp lift needs stronger per-rollout agents (harness engineering — tools/middleware/memory) and summarize-then-tournament-vote selection, not naive best-of-N. Those are the next builds.
The harness itself is the deliverable
The harness runs end-to-end with zero errors and full per-test breakdown — joule-bench can now produce real SWE-bench Verified numbers. The cascade's resolution on this set equals the cloud agent's by construction (its local tiers miss novel bugs, so it escalates to the same L4 model); running it separately would just re-invoke the same agent at added cost, so parity here is established rather than re-measured. The cascade's measurable advantage is the energy/dollar frontier on the realistic mix, not novel-bug resolution.