open-standards/eoc/docs/bench-v0.md

# EOC Benchmark v0 — `joules-per-MT-Bench-point`

> Status: draft v0. Numbers in the reference table below are placeholders
> until a live deployment is benchmarked end-to-end.
> License: text CC-BY-4.0, harness code Apache-2.0.

## 1. Why a joules-per-point metric

Every other benchmark in the LLM industry answers a single question:
*how good is the answer?* MMLU, MT-Bench, ARC, HumanEval, GPQA — all of
them measure quality on a fixed input distribution and ignore the energy
it took to produce the answer. That made sense when the only inference
substrate was a transformer running on a GPU and the only knob was
"more parameters". It does not make sense for an Energy-Optimised
Compute (EOC) deployment, where the substrate is a four-stage cascade
(cache → KV → graph → neural) and the explicit goal is to answer as
many queries as possible at the cheaper stages.

In that setting, the question that matters is:

> **How many joules did it take to earn one MT-Bench point?**

A cascade that scores 7.2 on MT-Bench at 14 J/correct is doing
qualitatively different work than a 7.2 GPU-baseline that burns 480
J/correct. The benchmark exists so a deployment can show that
difference rather than claim it.

The v0 formula is:

```
joules-per-MT-Bench-point =
    total_joules / (correct_answers × mt_bench_score_per_correct)
```

`total_joules` is the wall-time energy sum reported by the cascade for
the suite, in joules. `correct_answers` is the count of cases the
harness scored as a pass. `mt_bench_score_per_correct` is the suite's
own per-question weight; for the bundled MT-Bench-style set it is `1.0`,
so for v0 the metric reduces to **joules-per-correct**. The factor is
kept in the formula so that suites that adopt MT-Bench's 10-point judge
scoring later can plug it in without reshaping the dimensional
analysis.

A complementary metric, **stage-distribution**, reports the fraction of
cases resolved at each stage. A deployment that drives
joules-per-correct down by short-circuiting at cache and KV will see
that show up directly in the distribution. A deployment that scores
high by simply running every query through the neural stage will see
its joules-per-correct number balloon, regardless of how good the
final answers are.

## 2. What the harness actually does

The harness has four small crates:

| Crate                | Role                                                                                  |
| -------------------- | ------------------------------------------------------------------------------------- |
| `eoc-bench-runner`   | `BenchCase`, `BenchResult`, `BenchReport`; `run()`; `aggregate()`.                    |
| `eoc-bench-mt`       | 20 MT-Bench-style cases bundled as JSON (`data/cases.json`).                          |
| `eoc-bench-router`   | 30 router-shaped cases chosen to exercise all four cascade stages.                    |
| `eoc-bench` (binary) | CLI: `eoc-bench run --suite {mt,router,all}` and `eoc-bench compare`.                 |

`run()` walks the supplied cases through an `eoc_cascade::Cascade` one
by one. Sequential execution is intentional: parallel queries on shared
hardware pollute the joule attribution that the cascade hands back, and
the metric we care about is **energy per query**, not throughput. A
throughput-oriented variant is a future work item, not a v0 deliverable.

For each case the runner records:

* the response payload,
* which stage resolved it (`cache | kv | graph | neural`),
* the joule cost in microjoules,
* wall-clock latency in milliseconds,
* whether the response contained the expected substring (when the
  case has an `expected` field).

The default accuracy predicate is a case-insensitive substring match.
That is sufficient for the v0 case set, which uses unambiguous facts
("Paris", "Mars", "4") rather than open-ended writing tasks. Cases
designed to elicit creative output (`mt-writing-*`, `rt-neural-*`)
deliberately leave `expected` as `null`; they count towards joules and
latency but never towards accuracy.

`aggregate()` rolls a slice of `BenchResult`s up into a `BenchReport`
containing `case_count`, `total_joules`, `joules_per_correct`,
`accuracy_pct`, `latency_p50_ms`, `latency_p95_ms`, `latency_p99_ms`,
and a `stage_distribution` map. Percentiles use the nearest-rank
method (no interpolation) so the result is always one of the observed
latencies — this matches how operators read percentiles in production
dashboards.

## 3. The case sets

**MT (`eoc-bench-mt`).** Twenty cases drawn from the broad shape of
MT-Bench's eight categories — arithmetic, multi-step reasoning,
multi-turn dialogue, code, writing, factual recall, explanation,
summarisation. Multi-turn cases are concatenated with a literal
`Follow-up:` marker; the cascade sees the full conversation as a
single prompt, which is the same convention MT-Bench uses for turn
2 grading. Twenty is deliberately small: v0 is a debug-loop tool, not
a Goodhart target.

**Router (`eoc-bench-router`).** Thirty cases sized so a well-warmed
cascade hits all four stages. Case ids carry a routing hint:

* `rt-cache-*` — six short, repeated prompts that should memoise after
  the first warmup pass.
* `rt-kv-*` — seven exact-key lookups (well-known constants, ports,
  ISO codes) the reference KV stage is seeded with.
* `rt-graph-*` — eight subject–predicate–object questions the
  reference triple store can answer.
* `rt-neural-*` — nine open-ended generation prompts no pre-populated
  store can answer.

The hints are advisory. The harness measures *where the cascade
actually resolved each case*. A deployment that lands an
`rt-graph-*` case at `neural` because its triple store was not warmed
sees that show up in the stage distribution, and the joules-per-correct
number reflects the cost.

## 4. The reference cascade the CLI ships with

`eoc-bench run` instantiates the four-stage cascade from `eoc-rs`
using deliberately small fixture data so the CLI works out of the
box. The LRU cache has a 1024-entry capacity. The KV stage is seeded
with the same constants that the router suite's `rt-kv-*` cases ask
about. The graph stage holds the triples the `rt-graph-*` cases need.
The neural stage uses the `EchoBackend` from `eoc-neural` with an
estimated cost of 50 J per inference — well above realistic GPU cost
for any single token, but a useful default for visualising the
cost spread in the report. Production deployments substitute their
own wiring; the `run` function in `eoc-bench-runner` accepts any
`eoc_cascade::Cascade`.

## 5. Reference numbers

These are placeholders. They will be replaced as soon as the harness
runs against a live deployment.

| Suite     | Cases | Accuracy %                    | Joules/correct                | p50 / p95 / p99 (ms)                                   | Stage mix (cache / kv / graph / neural)                       |
| --------- | ----- | ----------------------------- | ----------------------------- | ------------------------------------------------------ | -------------------------------------------------------------- |
| mt        | 20    | _to be filled when running against a live deployment_ | _to be filled when running against a live deployment_ | _to be filled when running against a live deployment_ | _to be filled when running against a live deployment_         |
| router    | 30    | _to be filled when running against a live deployment_ | _to be filled when running against a live deployment_ | _to be filled when running against a live deployment_ | _to be filled when running against a live deployment_         |
| all       | 50    | _to be filled when running against a live deployment_ | _to be filled when running against a live deployment_ | _to be filled when running against a live deployment_ | _to be filled when running against a live deployment_         |

Targets we publish v0 results against:

* `joules-per-correct` ≤ **5 J** on the router suite for a well-warmed
  cascade (most cases resolve at cache/KV/graph).
* `joules-per-correct` ≤ **25 J** on the MT suite when wired against a
  small open-weights neural backend on commodity GPU.
* `latency_p95_ms` ≤ **250 ms** on the router suite end-to-end.
* `accuracy_pct` ≥ **80%** on the union of scored cases.

These thresholds are aspirational, not contractual. A deployment is
free to publish lower numbers and explain why; the metric is what
matters, not the threshold.

## 6. Reproducibility

The harness is fully embedded — the case JSON is `include_str!`'d at
build time, the path dependencies on `eoc-rs` are pinned in the root
`Cargo.toml`, and the toolchain is pinned via the workspace's
`rust-toolchain.toml` to Rust 1.95.0 edition 2024.

### Command lines

Build (default features include `eoc-rs`):

```
cargo build --release -p eoc-bench
```

Run a suite and capture a report:

```
target/release/eoc-bench run --suite mt    > mt-baseline.json
target/release/eoc-bench run --suite router > router-baseline.json
target/release/eoc-bench run --suite all    > all-baseline.json
```

Compare two reports:

```
target/release/eoc-bench compare \
    --baseline all-baseline.json \
    --candidate all-candidate.json
```

`compare` prints a small JSON object with the three deltas operators
care about:

* `joules_per_correct_pct` — percent change in J/correct (negative is
  better).
* `latency_p95_pct` — percent change in p95 latency (negative is
  better).
* `accuracy_pp_delta` — absolute change in accuracy percentage points
  (positive is better).

### BenchReport JSON schema

The `BenchReport` type serialises as:

```json
{
  "case_count": 20,
  "total_joules": 0.0,
  "joules_per_correct": 0.0,
  "accuracy_pct": 0.0,
  "latency_p50_ms": 0,
  "latency_p95_ms": 0,
  "latency_p99_ms": 0,
  "stage_distribution": {
    "cache": 0,
    "kv": 0,
    "graph": 0,
    "neural": 0
  }
}
```

Field semantics:

| Field                  | Type                | Meaning                                                                 |
| ---------------------- | ------------------- | ----------------------------------------------------------------------- |
| `case_count`           | `usize`             | Number of cases in the run.                                             |
| `total_joules`         | `f64`               | Sum of stage-reported energy, expressed in joules.                      |
| `joules_per_correct`   | `f64`               | `total_joules / correct_answers`. `Infinity` if no answers were correct. |
| `accuracy_pct`         | `f64`               | Percentage of *scored* cases that passed. Cases with no expected answer are excluded from this number. |
| `latency_p50_ms`       | `u64`               | Nearest-rank p50 latency across all cases.                              |
| `latency_p95_ms`       | `u64`               | Nearest-rank p95 latency.                                               |
| `latency_p99_ms`       | `u64`               | Nearest-rank p99 latency.                                               |
| `stage_distribution`   | `map<string,usize>` | Counts keyed by stable stage id (`cache`, `kv`, `graph`, `neural`).      |

### Diffability

Reports are intended to land in a repository as text-diffable JSON.
The aggregator is deterministic: same input cases plus the same cascade
configuration produces byte-identical reports. `serde_json::to_string_pretty`
emits stable key order because the underlying maps are `BTreeMap`s.

## 7. Honest limitations

This is v0. The following are known and tracked:

1. **The bundled neural backend is `EchoBackend`.** It does not produce
   correct answers for open-ended cases; its job is to exercise the
   joule attribution pipe and let the runner end-to-end. Real numbers
   require wiring a real backend (`llama.cpp`, vLLM, an inference
   provider) at the `NeuralBackend` trait surface.

2. **Accuracy scoring is substring-based.** That works for the v0
   case set because the expected answers are short and unambiguous.
   v1 should support pluggable judges, including LLM-as-judge for the
   writing/reasoning categories (matching MT-Bench's pipeline).

3. **The case sets are small.** Twenty MT cases and thirty router
   cases are enough to debug a cascade, not enough to publish a
   leaderboard against. v1 will adopt the full 80-question MT-Bench
   set and add a router suite of at least 250 cases.

4. **Joule readings come from the cascade, not from independent
   hardware counters.** The cascade itself trusts its meter (the
   default is a `StubCounter`). Deployments that need defensible
   numbers must wire `eoc-meter` against RAPL / NVML / `powermetrics`
   and re-run.

5. **`mt_bench_score_per_correct = 1.0`.** The MT-Bench original uses a
   10-point judge score per question. v0 collapses that to a binary
   pass/fail so the harness has no dependency on a judging model. v1
   will lift this restriction.

The point of shipping v0 anyway is that the *shape* of the metric — a
joule denominator times an accuracy numerator, with a stage
distribution alongside — is what makes the cascade visible. The
absolute numbers will get sharper. The shape will not.

## 8. License

Harness code (every Rust file under `eoc-bench/`): Apache-2.0.
This document: CC-BY-4.0. Bundled case JSON: CC-BY-4.0.