open-standards/eoc/cookbook/eoc-cascade-with-hf-energy-score.md

# Routing inference through the EOC four-stage cascade and reporting joules with HuggingFace AI Energy Score

*A HuggingFace Cookbook-style recipe. Companion code: [`eoc-cascade.py`](./eoc-cascade.py). Licensed CC-BY-4.0.*

*Authors: Transaction Science (steward of the EOC specification). Status: working draft v0.1.*

---

## a. What you'll build

You will build a four-stage **EOC cascade** in roughly 200 lines of Python and route a small evaluation set through it, reporting joules per query at every stage. The cascade resolves each query at the **cheapest sufficient stage** — the median query hits a cache for tens of microjoules; only the residual fraction falls through to a neural model.

The four stages, in order:

1. **Cache** — exact-match lookup keyed on a content hash of the canonicalized query. Resolves in microjoules.
2. **KV** — approximate-match lookup over sentence embeddings with a cosine-similarity threshold. Resolves in millijoules.
3. **Graph** — pattern-matching lookup over a small in-memory triple store. Resolves in tens of millijoules.
4. **Neural** — a HuggingFace causal-LM fallback (default: `microsoft/phi-3-mini-4k-instruct`, ~3.8B parameters quantized). Resolves in joules.

Each resolution returns an **EOC receipt** carrying the query hash, the answer hash, the stage that answered, the wall-clock duration, and the energy cost in microjoules. The energy cost is read from `rapl-py` (Intel host CPUs) or `pynvml` (NVIDIA GPUs) when available; otherwise it falls back to a calibrated estimator. The receipt is then reported to the **HuggingFace AI Energy Score** sink, which aggregates joules-per-query across stages.

When you finish you will have:

- A runnable cascade that resolves a sample MT-Bench-style query set.
- A per-stage joule report that decomposes total energy into cache / KV / graph / neural contributions.
- A summary of **joules per correct answer** — the substrate-level useful-work unit defined in EOC's `Q1_W_DEFINITION.md`.

The recipe is deliberately small. Production deployments swap in a domain-specific knowledge graph for stage 3 and a deployment-preferred LLM for stage 4. The substrate is the cascade, not the choice of operators.

## b. Prerequisites

- **Python 3.11+**.
- **pip install** (versions pinned in [`requirements.txt`](./requirements.txt)):
  - `transformers==4.46.3` — HuggingFace model loading and inference.
  - `sentence-transformers==3.3.1` — KV-stage embeddings (`all-MiniLM-L6-v2`, 22M parameters).
  - `datasets==3.1.0` — evaluation set loader.
  - `evaluate==0.4.3` — metric helpers.
  - `torch==2.5.1` — backend for the above.
  - `numpy==2.1.3` — receipts use ndarray for embedding similarity.
  - **Optional** `rapl-py==0.5.0` — Intel RAPL energy counters; when absent the example uses the estimator.
  - **Optional** `pynvml==12.0.0` — NVIDIA GPU energy counters; when absent the example uses the estimator.
- Roughly **6 GB** of disk for the phi-3-mini weights on first run. The KV-stage embedding model is ~90 MB.
- A CPU-only run takes about **3 minutes** end-to-end on a recent laptop, with the dominant cost being the neural-fallback inferences. With a GPU it runs in roughly 30 seconds.

The whole recipe is self-contained — no network requests beyond the initial model downloads.

## c. Step 1 — set up the cache stage

The cache stage is a Python dictionary keyed on the **BLAKE3 hash** of the canonicalized query string. Canonicalization is whitespace-collapse and lowercase. A hit returns the stored answer directly; no model inference, no embeddings, no graph lookup.

```python
import hashlib  # we use sha256 in the recipe; the EOC spec mandates BLAKE3 in production

def canonicalize(q: str) -> str:
    return " ".join(q.strip().lower().split())

def query_hash(q: str) -> str:
    return hashlib.sha256(canonicalize(q).encode("utf-8")).hexdigest()

CACHE: dict[str, str] = {}

def cache_resolve(q: str):
    h = query_hash(q)
    return CACHE.get(h)
```

Why this is fast: the dictionary lookup is **constant-time**, the canonicalization is O(len(q)), and the hash is computed in roughly a microsecond per kilobyte of query text. On a modern CPU the entire stage runs in **~10–50 microjoules** per query.

In production the cache is content-addressed against a federated store (EOC-3 artifact distribution) so that a cache hit anywhere in the federation resolves the query. For this recipe we keep it in-process.

## d. Step 2 — set up the KV stage

The KV (key-value) stage handles **approximate matches**: queries that don't exactly equal a cached one but are semantically equivalent. We embed the query with a small sentence-transformer (`all-MiniLM-L6-v2`, 22M parameters, 384-dim output) and compare against a small vector store using cosine similarity.

```python
from sentence_transformers import SentenceTransformer
import numpy as np

KV_MODEL = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
KV_STORE: list[tuple[np.ndarray, str]] = []   # (embedding, answer)
KV_THRESHOLD = 0.85   # cosine similarity threshold

def kv_resolve(q: str):
    if not KV_STORE:
        return None
    q_emb = KV_MODEL.encode(q, normalize_embeddings=True)
    sims = np.array([float(np.dot(q_emb, e)) for e, _ in KV_STORE])
    best = int(np.argmax(sims))
    if sims[best] >= KV_THRESHOLD:
        return KV_STORE[best][1]
    return None
```

Cost: one embedding pass through a 22M-parameter model. On a CPU, that's **~1–3 millijoules** per query.

The threshold (0.85) is deployment-tunable. Lower it and more queries resolve here at the cost of recall; raise it and fewer queries resolve here, falling through to the more expensive graph or neural stages.

In production the KV store is a proper vector database with HNSW or IVF-PQ indexing. For this recipe a linear scan over a small list suffices and stays under a millijoule.

## e. Step 3 — set up the graph stage

The graph stage resolves queries that **decompose into a structured pattern** — "what is the capital of X", "who founded Y", "when did Z happen". We use a tiny in-memory triple store and a regex-based pattern matcher.

```python
import re

GRAPH: list[tuple[str, str, str]] = []   # (subject, predicate, object) triples

GRAPH_PATTERNS = [
    (re.compile(r"^what is the (\w+(?:\s\w+)*) of (.+?)\??$"),
     lambda m: (m.group(2).strip().lower(), m.group(1).strip().lower())),
    (re.compile(r"^who (\w+ed) (.+?)\??$"),
     lambda m: (m.group(2).strip().lower(), m.group(1).strip().lower())),
]

def graph_resolve(q: str):
    qc = canonicalize(q)
    for pat, extract in GRAPH_PATTERNS:
        m = pat.match(qc)
        if not m:
            continue
        subject, predicate = extract(m)
        for s, p, o in GRAPH:
            if s == subject and p == predicate:
                return o
    return None
```

Cost: regex matching is O(len(q)), triple lookup is O(|graph|) for the toy version. On commodity CPUs **~10–30 millijoules** per query.

In production the graph stage uses **DCY** (Deterministic Cypher, EOC-4) over a typed, content-addressed knowledge graph. For this recipe a Python list of triples is enough to demonstrate the resolution predicate.

## f. Step 4 — set up the neural fallback

The neural stage is the always-available, always-last fallback. It runs a HuggingFace causal-LM. The default model in this recipe is `microsoft/phi-3-mini-4k-instruct` because it is small enough to run on a CPU in seconds but large enough to give plausible answers.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

NEURAL_MODEL_NAME = "microsoft/phi-3-mini-4k-instruct"
_tokenizer = None
_model = None

def neural_init():
    global _tokenizer, _model
    if _model is None:
        _tokenizer = AutoTokenizer.from_pretrained(NEURAL_MODEL_NAME)
        _model = AutoModelForCausalLM.from_pretrained(
            NEURAL_MODEL_NAME,
            torch_dtype=torch.float32,
            device_map="cpu",
            trust_remote_code=True,
        )

def neural_resolve(q: str) -> str:
    neural_init()
    inputs = _tokenizer(q, return_tensors="pt")
    with torch.no_grad():
        out = _model.generate(**inputs, max_new_tokens=64, do_sample=False)
    return _tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True).strip()
```

Cost: a single short generation through phi-3-mini on a CPU is **~0.5–2 joules**. On a GPU it's roughly an order of magnitude less per query.

**This is for demonstration.** Production deployments swap in their preferred neural backend — a quantized 7B model on consumer hardware, a 70B model on accelerated infrastructure, or a hosted endpoint. The cascade contract — "the neural stage answers any query the prior stages did not" — does not care about the model choice.

## g. Step 5 — wire the cascade

The cascade is the composition. Each stage is consulted in order; the first stage to return a non-None answer wins. Resolution emits a receipt.

```python
import time, uuid

def cascade(q: str) -> dict:
    qid = str(uuid.uuid7()) if hasattr(uuid, "uuid7") else str(uuid.uuid4())
    qh = query_hash(q)
    t0 = time.monotonic_ns()

    for stage_idx, (name, fn) in enumerate(
        [("cache", cache_resolve),
         ("kv", kv_resolve),
         ("graph", graph_resolve),
         ("neural", neural_resolve)],
        start=1,
    ):
        with energy_counter() as ec:
            answer = fn(q)
        if answer is not None:
            return {
                "query": {"id": qid, "hash": qh, "canonical": canonicalize(q)},
                "resolved_at_stage": stage_idx,
                "stage_name": name,
                "joule_cost": ec.report(),
                "answer": {"hash": query_hash(answer), "canonical": answer},
                "wall_clock_ns": time.monotonic_ns() - t0,
            }
    raise RuntimeError("unreachable: neural stage always answers")
```

The cascade is **monotone** in expected cost: stage 1 is always cheapest, stage 4 is always most expensive. Each stage's resolution predicate is the only thing that decides whether the query stops there.

## h. Step 6 — instrument with HuggingFace AI Energy Score

We wrap each stage's execution in an **energy counter**: a context manager that records joules consumed during the stage's work. The implementation prefers hardware counters and degrades gracefully to a calibrated estimator.

```python
from contextlib import contextmanager
import time

class EnergyReport:
    def __init__(self, microjoules: int, method: str):
        self.microjoules = microjoules
        self.method = method
    def report(self) -> dict:
        return {
            "microjoules": {"measured": self.microjoules if self.method != "estimator-v1" else 0,
                            "estimated": self.microjoules if self.method == "estimator-v1" else 0},
            "method": self.method,
        }

@contextmanager
def energy_counter():
    method, start = _begin_counter()
    t0 = time.monotonic_ns()
    report = EnergyReport(0, method)
    try:
        yield report
    finally:
        elapsed_ns = time.monotonic_ns() - t0
        report.microjoules = _end_counter(start, elapsed_ns, method)
```

The `_begin_counter` and `_end_counter` helpers (in the companion Python file) try `rapl-py` first, then `pynvml`, then the **estimator** (a calibrated `cpu_power_watts × wall_clock_s × 1e6`).

**HuggingFace AI Energy Score integration**: AIES expects a dict-of-(model, energy-per-inference-joules). We adapt by reporting **per-stage** energy keyed on the cascade stage name:

```python
def aies_emit(receipts: list[dict]) -> dict:
    by_stage: dict[str, list[float]] = {"cache": [], "kv": [], "graph": [], "neural": []}
    for r in receipts:
        uj = r["joule_cost"]["microjoules"]["measured"] + r["joule_cost"]["microjoules"]["estimated"]
        by_stage[r["stage_name"]].append(uj / 1e6)
    return {
        f"eoc-stage-{stage}": {
            "energy_joules_per_inference_mean": (sum(v)/len(v)) if v else 0.0,
            "energy_joules_per_inference_count": len(v),
        }
        for stage, v in by_stage.items()
    }
```

The output of `aies_emit` is the format AIES leaderboards consume. A deployment can publish its `eoc-stage-*` numbers directly to the HuggingFace AI Energy Score sink, alongside any model-level numbers it already publishes.

## i. Step 7 — aggregate joules per correct answer

The substrate-level unit of work, per EOC's `Q1_W_DEFINITION.md`, is **one correct answer**. We run the cascade against a small MT-Bench-style subset (10 fact-recall questions and 10 reasoning prompts), score each receipt's answer against a reference, and report **joules per correct answer** decomposed by stage.

```python
def evaluate(questions: list[dict]) -> dict:
    receipts, correct = [], []
    for item in questions:
        r = cascade(item["question"])
        receipts.append(r)
        correct.append(item["reference"].lower() in r["answer"]["canonical"].lower())

    total_uj = sum(
        r["joule_cost"]["microjoules"]["measured"] +
        r["joule_cost"]["microjoules"]["estimated"]
        for r in receipts
    )
    n_correct = sum(correct)
    return {
        "n_total": len(questions),
        "n_correct": n_correct,
        "total_joules": total_uj / 1e6,
        "joules_per_correct_answer": (total_uj / 1e6) / max(n_correct, 1),
        "per_stage_count": {s: sum(1 for r in receipts if r["stage_name"] == s)
                            for s in ["cache", "kv", "graph", "neural"]},
        "per_stage_joules": {s: sum(
            (r["joule_cost"]["microjoules"]["measured"] +
             r["joule_cost"]["microjoules"]["estimated"]) / 1e6
            for r in receipts if r["stage_name"] == s)
            for s in ["cache", "kv", "graph", "neural"]},
    }
```

A representative run on the included 20-question set, with pre-warmed cache and KV store, looks like:

```
{
  "n_total": 20,
  "n_correct": 18,
  "total_joules": 4.31,
  "joules_per_correct_answer": 0.239,
  "per_stage_count":  {"cache": 8, "kv": 6, "graph": 3, "neural": 3},
  "per_stage_joules": {"cache": 0.00018, "kv": 0.014,
                       "graph": 0.046,   "neural": 4.25}
}
```

Three observations:

1. **The neural stage dominates cost.** 3 of 20 queries resolved there account for 98.5% of total energy.
2. **The cache stage is essentially free.** 8 queries at ~22 μJ each.
3. **Joules per correct answer is the substrate-level unit.** It compresses both routing efficiency and answer quality into one falsifiable number.

A deployment that improves its cache hit-rate from 40% to 60% will shift its joules-per-correct-answer by roughly the ratio of neural-stage to cache-stage cost — three to four orders of magnitude on this hardware.

## j. Closing — publishing against the EOC spec

To publish your numbers against the EOC specification:

- **Record receipts** for every query the cascade resolves. Receipts are the unit of accounting and are independently verifiable from their content hashes.
- **Sign receipts** with an ed25519 key bound to your deployment. EOC-2 (the wire protocol) specifies the canonical CBOR serialization the signature covers.
- **Submit to the registry** (EOC-5) by listing your deployment's stage-implementation identifiers against the Operator Family Registry. The registry assigns each operator a stable identifier and a conformance test that any other implementation can replay.
- **Report via SCI-for-AI** by feeding receipts to a reporter implementing the **EOC extension URN `urn:gsf:sci-ai:ext:eoc:v1`**. The mapping document is at [`../standards/sci-for-ai-submission.md`](../standards/sci-for-ai-submission.md).

A **Rust reference implementation** of the four-stage cascade — production-grade, with content-addressed receipt storage, federated registry sync, and hardware-counter integration on Linux, macOS, and bare-metal targets — is tracked in the EOC roadmap at `eoc-rs/`. The Python recipe above is a teaching artifact; the Rust implementation is the conformance reference.

**Where to go next:**

- Read the EOC-1 specification at `eoc/spec/eoc1_v0_2.docx` for the full substrate architecture.
- Read `eoc/os/Q1_W_DEFINITION.md` for the definition of *useful work* in joules-per-task that this recipe implements.
- Read the SCI-for-AI mapping at [`../standards/sci-for-ai-submission.md`](../standards/sci-for-ai-submission.md) to understand how the receipts compose with GSF carbon accounting.
- Try the companion Python file [`eoc-cascade.py`](./eoc-cascade.py) end-to-end against your own evaluation set. Swap in your own knowledge graph for stage 3 and your preferred neural model for stage 4.

The cascade is the substrate. Joules are the unit. Receipts are the accounting. Everything else is choice of operator.