joule-bench/tasks/real/config-defaults/reference/schema.py
David Charlot b8fbc0c253 Add a suite built to the measured distribution; local tiers score 0/6
`mix` over-represents L1-shaped work by 25-40x against real commit
history, so this adds `tasks/real` built to the distribution actually
measured: three tasks whose fix spans two source files, two single-file
rewrites of real logic, one small single-file fix. Ordinary bugs — a
sliding window that never prunes, a parser folding operators left to
right, a loader accepting misspelled keys, interval merging that assumes
sorted input, a retry loop that retries unretryable failures, a page
count dropping the last partial page.

The suite is sound: reference 6/6, noop 0/6.

The cascade's local tiers resolve **0 of 6**, against 4/5 on `mix`. None
of these bugs is a substitution, a cached repeat or a paraphrase, so L0,
L1 and L2 have nothing to offer and every task falls through in under a
second.

That is the result the composition analysis predicted and it bounds the
local claim precisely: the cheap tiers address a real but narrow class of
work, and that class is not what fills a commit log. The 7.6x advantage
on `mix` is a property of that suite's composition rather than a general
one.

What this does not touch is the cache, which is where the cascade's
economics actually live: every task here runs once, so L0 never fires. An
L4-enabled two-pass run over this suite would measure it — first pass
pays the cloud for all six, second should close them free — and has not
been run.

Building a suite we score zero on is the point. A benchmark that only
contains work we are good at measures nothing.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-11 00:29:11 -04:00

9 lines
122 B
Python

DEFAULTS = {
"host": "localhost",
"port": 8080,
"timeout": 30,
}
def known(key):
return key in DEFAULTS