9fbcc9e7c8
Found by running the suite rather than reading it. Each one is fatal to a different column, and together they explain the numbers in RESULTS.md. **Resolution was under-reported for every solver on Python tasks.** The baseline run — the one that proves the bug reproduces — leaves a `__pycache__` behind, and the graded run imported that stale bytecode, so a correct fix still failed to import. The `reference` solver applies the task's own known-good patch and is by construction the ceiling; it scored 4/5 on `mix`. A benchmark whose ceiling is not 100% is not measuring anything. Compiled-artifact caches are now discarded between the solver and the grade. **Every energy reading was measuring the meter.** One `read()` on the Apple model-based counter takes ~840 ms and reports the energy of its own sampling window, and `MeasuredMeter` bracketed each task between two of them. A 0 ms task read as 34 J. The counter is now sampled on a background thread and the meter reads an atomic, so a task shorter than the sampling window reads 0 and is flagged incomparable — which is true, and better than a fabricated number. **Idle draw was calibrated at about nine times reality.** The old calibration slept 300 ms between two 840 ms reads, then divided an energy delta covering ~2 s by 0.3 s: it reported **352.2 W** idle for a Mac. The corrected window gives **38.2 W**. The inflated figure was subtracted from every task, which is the real reason the published local-energy column read ~0. The meter's minimum window now reflects the sampler's period rather than the counter's declared resolution, so the noise guard tells the truth about which runs are comparable. Also adds an `agent` solver driving `joule-coded agent`: the full loop with tools and the verification gate, rather than `cascade`'s single-file tiers. That is the surface comparable to a Claude Code or Codex benchmark number, which measures a harness. RESULTS.md is marked superseded rather than deleted, with the three defects named, so the old figures are not quietly replaced. Co-Authored-By: Claude Opus 5 <[email protected]>
23 lines
536 B
JSON
23 lines
536 B
JSON
{
|
|
"total_instances": 2,
|
|
"submitted_instances": 2,
|
|
"completed_instances": 0,
|
|
"resolved_instances": 0,
|
|
"unresolved_instances": 0,
|
|
"empty_patch_instances": 2,
|
|
"error_instances": 0,
|
|
"completed_ids": [],
|
|
"incomplete_ids": [],
|
|
"empty_patch_ids": [
|
|
"pallets__flask-5014",
|
|
"psf__requests-1766"
|
|
],
|
|
"submitted_ids": [
|
|
"pallets__flask-5014",
|
|
"psf__requests-1766"
|
|
],
|
|
"resolved_ids": [],
|
|
"unresolved_ids": [],
|
|
"error_ids": [],
|
|
"schema_version": 2
|
|
}
|