cbcc27f1f7
The docs said a host with no per-process counter gets the token-count
estimate off macOS, but there ProcessSpanMeter falls back to detect_meter()
(the 28.6 W x wall estimator in this build), which it marks Whole, and grade()
kept that figure. A fallback span is the machine's figure, not the engine's,
so grade() now treats it as no reading and uses the rest (host_counter, else
the token estimate): one way of reading the host whichever engine ran, and
the docs hold. That is simpler than documenting a second host figure that
only the in-process engine would get.
JOULE_CODE_L3_SELF_CONTAINED could never take effect: the llama engine is
read at its exit, which the tree meter always marks (TreeGap::Exited), so no
reading was ever whole. The setting and its plumbing are gone and the docs
say the reading is a lower bound, always graded as an estimate.
A missing llama binary had become LlmError::Upstream("... exited 127"): the
binary is exec'd by the gating sh, whose failed exec reads as the model's own
failure. The binary is now resolved first (a path as given, else PATH; a
regular executable file) and one that cannot run is Unavailable, as a failed
spawn was, with nothing spawned.
The README says the oracle tests run only locally on Apple Silicon (CI tests
this workspace on Linux, where they take the fallback), and lines this branch
added name the capability, not a third-party engine.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
93 lines
4.9 KiB
Markdown
93 lines
4.9 KiB
Markdown
# L3 binding — JouleClaw model tier → Joule Code cascade
|
|
|
|
A consumer **binding example**: how to wire JouleClaw's **L3 (Model)** tier into
|
|
the Joule Code cascade. It is a standalone workspace, so the standard's reference
|
|
library keeps no JouleClaw dependency.
|
|
|
|
A coding request that misses the cheaper tiers (L0 cache, L1 lawful, L2 embed) is
|
|
generated by a local model at L3, producing a content-addressed `CodingOp` and a
|
|
joules-class `Receipt`.
|
|
|
|
## Three backends, same slot
|
|
|
|
All implement JouleClaw's `jouleclaw_llm_cheap::LlmBackend`, so they drop into
|
|
the same cascade stage:
|
|
|
|
- **`EchoBackend`** (from JouleClaw) — deterministic, model-free; used by the
|
|
unit tests so the wiring is verifiable with no model present.
|
|
- **`JouleclawRuntimeBackend`** (here) — runs the model **in-process on
|
|
JouleClaw's own from-scratch GGUF tensor runtime** (`jouleclaw-runtime`): no
|
|
external binary, no FFI. Tokenize → build the decode-step graph → compile
|
|
against the registered kernels (`jouleclaw-backend-reference` /
|
|
`-apple`) → autoregressive decode → detokenize, all in Rust the family wrote.
|
|
This is the **default** backend for the `run` binary.
|
|
- **`LlamaCompletionBackend`** (here) — drives a real on-device GGUF via
|
|
`llama-completion` (llama.cpp). Set `JOULE_CODE_L3_ENGINE=llama` to use it.
|
|
|
|
Either local path: no datacenter, no network. In production the cheapest capable
|
|
backend is chosen by `jouleclaw_dispatch::ModelBank`'s cost table.
|
|
|
|
## Run it against a local model
|
|
|
|
```bash
|
|
# Default: in-process on jouleclaw-runtime (the from-scratch tensor engine).
|
|
cargo run --release --bin run -- /path/to/model.gguf "Write a Rust function that reverses a string."
|
|
# or: JOULE_CODE_L3_MODEL=/path/to/model.gguf cargo run --release --bin run
|
|
|
|
# Use the llama.cpp shell-out backend instead:
|
|
JOULE_CODE_L3_ENGINE=llama cargo run --release --bin run -- /path/to/model.gguf "..."
|
|
```
|
|
|
|
The `run` binary generates with the local model, content-addresses the result as
|
|
a `CodingOp` (tier L3), and puts the energy of the inference on the receipt, read
|
|
as close to the work as the host allows and graded by how much of the inference
|
|
the reading covers:
|
|
|
|
- **In-process engine (default)** — metered on this process's own CPU and GPU
|
|
(`jouleclaw_energy::span::ProcessSpanMeter`): the CPU energy the kernel counts
|
|
for this process's threads, plus its GPU energy — an exact zero where it spent
|
|
no GPU time, the GPU rail where it had the GPU to itself. A span that covers the
|
|
whole inference is a hardware reading, `HwShunt` with `measured_uj` set. One
|
|
that covers part of it (the GPU shared with another process, so the CPU alone)
|
|
is an estimate of the whole, `Estimator`: the reading plus the host meter's
|
|
figure over the same window.
|
|
- **llama engine** — the subprocess is read on its own process tree
|
|
(`sandbox_meter::TreeMeter`) where the host counts each process's CPU energy
|
|
(macOS). That reading is never whole: it is the process's CPU only, not the GPU
|
|
the binary may offload to, and the process exits in its window, so its record
|
|
lacks its last time on a core — a lower bound. So it is always an estimate of
|
|
the whole, the reading plus the host meter's figure, never `HwShunt`.
|
|
- **No reading of the engine's own process** — no per-process counter, or, for
|
|
the in-process engine, this process's counters not live (the span then falls
|
|
back to a host meter, whose figure is the machine's, not the engine's): the
|
|
host meter's figure as it is. That is JouleClaw's IOReport counter where its
|
|
CPU energy rail is live (`HwShunt`), its load-average estimator where it is not
|
|
(macOS 27.0.1 for an unprivileged process, `Estimator`), and the token-count
|
|
estimate anywhere else (Linux and Windows included; this binding builds
|
|
`jouleclaw-energy` with the `ioreport` feature only).
|
|
|
|
The receipt has no coverage field, so a partial reading is never sealed as a
|
|
hardware reading of the whole. The printout names the engine, the host meter and
|
|
the IOReport status, and says how the figure was obtained.
|
|
|
|
Verified live: the llama.cpp path against a local 350M GGUF, and the in-process
|
|
from-scratch runtime against `ternary-bonsai-1.7b-q2_0` (qwen3, Q2_0) — real text
|
|
generated, a real `op_id`, the cascade closing at L3.
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
cargo test # EchoBackend path — no model required
|
|
|
|
# In-process from-scratch runtime against a real GGUF (opt-in):
|
|
JOULE_BONSAI=/path/to/ternary-bonsai-1.7b-q2_0.gguf \
|
|
cargo test --release --test runtime_l3 -- --ignored --nocapture
|
|
```
|
|
|
|
The `run` binary's meter tests check each engine's reading against an oracle
|
|
sharing no code with the meter — the process's own energy count, read from
|
|
inside it — and assert it only where this host counts each process's energy
|
|
(Apple Silicon macOS, the per-process counter live); elsewhere they check the
|
|
fallback. CI tests this workspace on Linux only, where they take the fallback,
|
|
so the oracle assertions are not in CI and run only locally: run
|
|
`cargo test -- --nocapture` on an Apple Silicon Mac to see them.
|