eda1628ec3
A solver-agnostic harness that grades coding-agent task success with an executable test oracle (tests fail before, pass after; pristine test files restored to prevent gaming) and pairs it with measured energy (jouleclaw IOReport, idle-subtracted, sub-window readings flagged unreliable), vendor-billed dollars, and token-estimated cloud energy. Solvers: reference, noop, claude, gemini, and cascade (the real joule_core engine via joule-coded). --passes shows L0 cache amortization. Headline (RESULTS.md): on a realistic operation mix, the cascade and frontier claude both resolve 100%, but the cascade costs ~10x less per task -- mechanical ops close locally at L1 for free, novel ops escalate to the same model once, and repeats are free from the L0 cache. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
20 lines
285 B
Python
20 lines
285 B
Python
import pytest
|
|
|
|
from calc import calc
|
|
|
|
|
|
def test_add():
|
|
assert calc(2, "+", 3) == 5
|
|
|
|
|
|
def test_divide():
|
|
assert calc(6, "/", 2) == 3.0
|
|
|
|
|
|
def test_divide_float():
|
|
assert calc(7, "/", 2) == 3.5
|
|
|
|
|
|
def test_unknown():
|
|
with pytest.raises(ValueError):
|
|
calc(1, "%", 2)
|