b8fbc0c253
`mix` over-represents L1-shaped work by 25-40x against real commit history, so this adds `tasks/real` built to the distribution actually measured: three tasks whose fix spans two source files, two single-file rewrites of real logic, one small single-file fix. Ordinary bugs — a sliding window that never prunes, a parser folding operators left to right, a loader accepting misspelled keys, interval merging that assumes sorted input, a retry loop that retries unretryable failures, a page count dropping the last partial page. The suite is sound: reference 6/6, noop 0/6. The cascade's local tiers resolve **0 of 6**, against 4/5 on `mix`. None of these bugs is a substitution, a cached repeat or a paraphrase, so L0, L1 and L2 have nothing to offer and every task falls through in under a second. That is the result the composition analysis predicted and it bounds the local claim precisely: the cheap tiers address a real but narrow class of work, and that class is not what fills a commit log. The 7.6x advantage on `mix` is a property of that suite's composition rather than a general one. What this does not touch is the cache, which is where the cascade's economics actually live: every task here runs once, so L0 never fires. An L4-enabled two-pass run over this suite would measure it — first pass pays the cloud for all six, second should close them free — and has not been run. Building a suite we score zero on is the point. A benchmark that only contains work we are good at measures nothing. Co-Authored-By: Claude Opus 5 <[email protected]>
45 lines
958 B
Python
45 lines
958 B
Python
import pytest
|
|
from retry import call_with_retry, Permanent
|
|
|
|
|
|
def test_succeeds_first_time():
|
|
assert call_with_retry(lambda: 7) == 7
|
|
|
|
|
|
def test_retries_then_succeeds():
|
|
calls = {"n": 0}
|
|
|
|
def flaky():
|
|
calls["n"] += 1
|
|
if calls["n"] < 3:
|
|
raise ValueError("transient")
|
|
return "ok"
|
|
|
|
assert call_with_retry(flaky) == "ok"
|
|
assert calls["n"] == 3
|
|
|
|
|
|
def test_permanent_is_not_retried():
|
|
calls = {"n": 0}
|
|
|
|
def bad():
|
|
calls["n"] += 1
|
|
raise Permanent("nope")
|
|
|
|
with pytest.raises(Permanent):
|
|
call_with_retry(bad)
|
|
assert calls["n"] == 1, "a permanent failure must not be retried"
|
|
|
|
|
|
def test_delays_are_exponential():
|
|
waited = []
|
|
calls = {"n": 0}
|
|
|
|
def flaky():
|
|
calls["n"] += 1
|
|
if calls["n"] < 4:
|
|
raise ValueError("transient")
|
|
return "ok"
|
|
|
|
call_with_retry(flaky, attempts=4, sleep=waited.append)
|
|
assert waited == [1, 2, 4]
|