joule-bench/tasks/mix/const
Repository files (latest commit first)
Filename Latest commit message Latest commit date
David Charlot eda1628ec3 joule-bench: Resolution x energy/cost coding-agent benchmark
A solver-agnostic harness that grades coding-agent task success with an
executable test oracle (tests fail before, pass after; pristine test files
restored to prevent gaming) and pairs it with measured energy (jouleclaw
IOReport, idle-subtracted, sub-window readings flagged unreliable), vendor-billed
dollars, and token-estimated cloud energy. Solvers: reference, noop, claude,
gemini, and cascade (the real joule_core engine via joule-coded). --passes shows
L0 cache amortization.

Headline (RESULTS.md): on a realistic operation mix, the cascade and frontier
claude both resolve 100%, but the cascade costs ~10x less per task -- mechanical
ops close locally at L1 for free, novel ops escalate to the same model once, and
repeats are free from the L0 cache.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-22 07:18:36 -04:00
..
before joule-bench: Resolution x energy/cost coding-agent benchmark 2026-06-22 07:18:36 -04:00
reference joule-bench: Resolution x energy/cost coding-agent benchmark 2026-06-22 07:18:36 -04:00
task.toml joule-bench: Resolution x energy/cost coding-agent benchmark 2026-06-22 07:18:36 -04:00