119b3afa70
The best-of-k frontier previously reported only the oracle ceiling (resolved-if-any), which isn't realizable: you'd need the hidden grader to know which rollout to ship. This adds the selection half — a no-leak BFT tournament vote that actually picks ONE candidate per instance — so the frontier reports the resolution a realizable selector achieves alongside the ceiling, at identical cost (all k rollouts run either way). - TaskRun carries a stable patch_digest (FNV-1a over the changed non-test files vs the pristine tree), computed before grading so it never sees the oracle verdict. - select_consensus: plurality vote over candidate digests (ties → earliest rollout); abstains when nothing patched. Maps onto agent-joule's lockstep-quic BFT fleet — N metered workers vote, consensus ships one. - TestTimePoint/test_time_frontier + render gain selected_resolved; the oracle−consensus gap is pure selection-quality signal. - Mirrored in the SWE-bench harness (consensus_pick over per-rollout patches) so real Verified runs report oracle vs consensus. Tests: plurality pick, the honest case where consensus MISSES an oracle win (lone correct patch outvoted), abstention, frontier monotonicity (selected ≤ oracle), digest stability. 5/5 pass; end-to-end smoke renders. Co-Authored-By: Claude Opus 4.8 <[email protected]>
103 lines
5.3 KiB
Markdown
103 lines
5.3 KiB
Markdown
# SWE-bench Verified — external parity check
|
||
|
||
This runs a solver over a subset of [SWE-bench Verified](https://www.swebench.com/)
|
||
(500 human-validated GitHub issues) through the **official `swebench` Docker
|
||
evaluation harness** — the externally-recognized coding-agent benchmark. It
|
||
complements the energy/cost frontier in [`../RESULTS.md`](../RESULTS.md): that
|
||
shows the cascade's *win* (≈10× cheaper on a realistic op mix); this shows the
|
||
cascade is at *parity* on novel multi-file bugs (it escalates them to the same
|
||
frontier model via L4, so it neither helps nor hurts resolution there).
|
||
|
||
## Setup (macOS, Apple Silicon)
|
||
|
||
```sh
|
||
open -a OrbStack # a Docker daemon (fast; emulates linux/amd64)
|
||
python3.11 -m venv ~/.swebench-venv # swebench needs Python 3.10+
|
||
~/.swebench-venv/bin/pip install swebench datasets
|
||
~/.swebench-venv/bin/python swebench/run_swebench.py --solver claude \
|
||
--instances psf__requests-5414 pylint-dev__pylint-7080
|
||
```
|
||
|
||
The SWE-bench prebuilt images are `linux/amd64`; on Apple Silicon they build/run
|
||
under emulation (slower, occasionally flaky). 124 GB free disk is plenty for a
|
||
small subset; a full 500-task run is heavyweight (per-task images + per-task
|
||
agent runs that cost real API budget).
|
||
|
||
## What it measures, honestly
|
||
|
||
A run produces resolution rate (the harness builds the task image, applies the
|
||
solver's source patch + the hidden test patch, and runs FAIL_TO_PASS +
|
||
PASS_TO_PASS) paired with the solver's dollar cost.
|
||
|
||
**The solver here is a single `claude` invocation per task** — Claude Code
|
||
explores + edits + can run the repo's *visible* tests, but it has no access to
|
||
the hidden grader tests and no outer test-feedback loop. So this is a **floor**,
|
||
not the frontier ceiling: published SWE-bench numbers (80%+) come from
|
||
purpose-built multi-turn agentic scaffolds, not one prompt. The number's
|
||
absolute value is a property of the scaffold, not the model.
|
||
|
||
## Sample (9 instances, real Docker eval, aarch64/OrbStack)
|
||
|
||
```
|
||
instance resolved $ cost
|
||
psf__requests-1921 ✗ no $0.299 (wrong fix; broke 24 tests)
|
||
psf__requests-5414 ✓ yes $0.538
|
||
psf__requests-6028 (empty) $0.260 (agent errored, no patch)
|
||
pylint-dev__pylint-7080 ✓ yes $2.893
|
||
pallets__flask-5014 ✓ yes $0.310
|
||
psf__requests-1766 ✗ no $0.425
|
||
psf__requests-2931 ✓ yes $1.327
|
||
pylint-dev__pylint-4970 ✗ no $0.527
|
||
pylint-dev__pylint-7277 ✓ yes $0.759
|
||
─────────────────────────────────────────────────────
|
||
RESOLVED 5/9 = 56% total $7.34 $1.47/resolved
|
||
```
|
||
|
||
(Run as two batches via `run_swebench.py`; the second 5-instance batch resolved
|
||
3/5 at $1.12/resolved.) Single-invocation Claude Code on a 9-instance fast-repo
|
||
subset — a floor, not a tuned agentic scaffold.
|
||
|
||
## Metered test-time compute (`--rollouts N`)
|
||
|
||
`--rollouts N` runs N independent rollouts per instance and reports the best-of-k
|
||
frontier paired with cumulative dollars — the resolution × $ curve of test-time
|
||
compute, **metered** (no other harness measures the cost of the lift). Each k now
|
||
reports two resolution numbers:
|
||
|
||
- **oracle** — resolved if *any* of the first k rollouts resolves. The ceiling of
|
||
any selection method; not realizable (you'd need the hidden grader to know
|
||
which rollout to ship).
|
||
- **consensus** — the **no-leak BFT tournament vote** (`consensus_pick`) actually
|
||
selects *one* candidate per instance: the patch the most rollouts agree on
|
||
(plurality over normalized diffs, ties → earliest rollout), never consulting
|
||
the grader. This is the resolution a *realizable* selector achieves. The cost
|
||
is identical to oracle — all k rollouts run either way — so the oracle−consensus
|
||
gap is pure selection-quality signal, not a cost trade.
|
||
|
||
```
|
||
k oracle consensus cum $
|
||
1 2/3 2/3 2.18
|
||
2 2/3 2/3 4.19
|
||
3 2/3 2/3 6.24
|
||
```
|
||
|
||
Honest reading: this sample's failure was *deterministic*, so even the oracle
|
||
ceiling stays flat while cost grows 3× — metering lets you *see* test-time compute
|
||
isn't paying off and stop. When rollouts do disagree, consensus is what closes the
|
||
oracle gap without peeking at the grader: independent metered workers vote and ship
|
||
the agreed patch (this maps directly onto agent-joule's lockstep-quic BFT fleet —
|
||
the vote is real consensus, not a heuristic). Realizing the literature's ~7pp lift
|
||
further needs **stronger per-rollout agents** (harness engineering —
|
||
tools/middleware/memory, e.g. the new `recall` long-term-memory tool in the L3
|
||
loop). The selector and its metering are now built; the remaining lever is agent
|
||
strength per rollout.
|
||
|
||
## The harness itself is the deliverable
|
||
|
||
The harness runs end-to-end with zero errors and full per-test breakdown —
|
||
joule-bench can now produce real SWE-bench Verified numbers.
|
||
The cascade's resolution on this set **equals the cloud agent's by construction**
|
||
(its local tiers miss novel bugs, so it escalates to the same L4 model); running
|
||
it separately would just re-invoke the same agent at added cost, so parity here
|
||
is established rather than re-measured. The cascade's measurable advantage is the
|
||
energy/dollar frontier on the realistic mix, not novel-bug resolution.
|