7a59f0ffc5
Brings over the uncommitted agent-mei work (Mac checkout, Sep 22) plus box follow-ups (Oct 17: agent_mei action namespace in keymap/zed.rs, MEI_BRIDGE.md, RENAME_AGENT_MEI.md, unmetered receipts report 0 uJ). joule_core, JOULE_CODE_* and JouleDB names unchanged. *.bak.* scratch files excluded.
74 lines
7.3 KiB
Markdown
74 lines
7.3 KiB
Markdown
# Fruition Audit — what's live vs aspirational
|
||
|
||
> Generated 2026-06-14. Skeptical pass: the project claims "live-verified" often — this checks the code, not the claims. file:line against audited HEAD.
|
||
|
||
## Bottom line
|
||
|
||
The crate is **substantially real, not a Potemkin demo.** The cascade (L0→L4 stop-at-first-close), both JCP gates (cascade-tier + agent-tool), lifecycle hooks (PreTool/PreMerge), the persistent sealed ledger, git-worktree sessions, the control protocol + MCP bridge, FIM edit-prediction, and the conformance harness are all genuinely wired, compile (11 MB rlib built today), and covered by **non-trivial unit tests with real assertions** (~270 across 13 files). Real crypto/energy/cascade work is delegated to the `open-standards` git deps, not faked.
|
||
|
||
The honest gaps are narrow and mostly disclosed:
|
||
|
||
1. **On-device tiers (L2/L3), the codebase index, and FIM are never CI-verified** — every real-model test is `#[ignore]`'d behind `JOULE_CODE_TEST_MODEL`. CI proves the plumbing (prompt-building, cleanup, vector math, cache round-trips), never that a model produces a usable edit.
|
||
2. **`author: [0u8;32]` is hardcoded on every emitted `CodingOp`** despite a real identity existing — ops are content-addressed but anonymous.
|
||
3. **L0 cache keys on intent only** (`prior` empty) — blind to buffer divergence.
|
||
4. **Sandbox sealing + MCP are best-effort**, dependent on `sandbox-mcp` on PATH; absent → edits silently apply unsealed.
|
||
5. **PostTool/SessionEnd hooks defined but never fired**; the L4 receipt is token-*estimated*, never measured (honestly tagged).
|
||
6. **`JOULE_CODE.md` is partly stale** — describes deps as `path = ...` (now git deps pinned to `4132daa`).
|
||
|
||
## Incomplete markers
|
||
|
||
No `unimplemented!`/`todo!`/`// TODO`/`// FIXME`/`// HACK` in production paths. `panic!`s are all in tests. "no-op"/"best-effort"/"placeholder" hits are honest descriptors. Load-bearing ones:
|
||
|
||
| Marker | Location | Load-bearing? |
|
||
|---|---|---|
|
||
| `author: [0u8; 32]` "per-run placeholder" | `agent_mei_panel.rs:3181, 3252` | **Yes** — anonymizes every op |
|
||
| deterministic-seed identity fallback | `:4005-4018` | Minor — only when keyfile unwritable |
|
||
| "inert pre/post/inv" contracts | `methodology.rs:87` | Honest scoping — structural frame_check only |
|
||
| PostTool/SessionEnd "the place to record" | `hooks.rs:25,29` | **Defined, never fired** |
|
||
| "best-effort: write failure is logged" | `agent_mei_panel.rs:684,737,827,969` | Honest — memory degrades, never errors |
|
||
|
||
## Silent/empty paths
|
||
|
||
- **`author: [0u8;32]`** — `captured_op` `:3181`, `build_coding_op` `:3252`. `op_id()` is real BLAKE3, but the author is zeroed; `session_identity` `:3994` builds a real ed25519 key used for ledger/methodology seals but **not** stamped into op cargo. The single most material gap vs "signed transition" claims.
|
||
- **L0 `prior` always empty** — `:676,1133,1625,2136,2565`. Cache key = intent string only.
|
||
- **Load-bearing `let _ =` discards** (vs the repo's CLAUDE.md rule): `:633` (`adopt_from_file`), `:953` (`cite_op`), `:978/981` (persistence writes), `:2893` (`record_primary_op`). The rest (~35) are test cleanup.
|
||
- **`session_receipt`** `:3107`: `sequence:0, nonce:[0u8;8]` hardcoded — fine for a session-summary receipt, not per-op (disclosed).
|
||
|
||
## Tests
|
||
|
||
**5 `#[ignore]`'d tests**, all gated on a real model/server:
|
||
|
||
| Test | File:line | Proves | CI |
|
||
|---|---|---|---|
|
||
| `l3_probe_simple_vs_harness` | `agent_mei_panel.rs:4861` | L3 produces usable edits | never |
|
||
| `harness_probe_one` | `:4910` | a code model emits a valid `<<<FILE>>>` edit | never |
|
||
| `codebase_index_retrieves_relevant_file_with_real_model` | `:4933` | semantic retrieval | never |
|
||
| `embeds_separate_paraphrase_from_unrelated` | `:5114` | L2 embedding separation | never |
|
||
| `connects_to_a_real_mcp_server_and_lists_tools` | `mcp_client.rs:219` | MCP bridge to a real server | never |
|
||
|
||
**Non-ignored tests are load-bearing, not smoke** — e.g. `ledger_chain_round_trips_through_a_real_file_and_detects_tampering` (real I/O + seal + tamper), `sealed_ledger_receipt_binds_op_to_one_identity` (wrong-key + tampered-op rejection), the `cascade_gate` budget/grant/lifetime trio, `tool_gate` deny tests, `hooks` deny-wins + point-isolation, the `sessions` real-`git` worktree merge tests, and the conformance test's `report.pass > 0` vacuity guard. The gap is entirely the on-device-model layer.
|
||
|
||
## Claim vs reality
|
||
|
||
| Feature | Claimed | Actual | Evidence | Gap |
|
||
|---|---|---|---|---|
|
||
| **L0 cache** | session + cross-session memory | **Live** | `submit_prompt:1624`; rehydrate `:676` | Keys on intent only — blind to buffer |
|
||
| **L1 deterministic** | literal substitution, no model | **Live** | `:1834`; `parse_substitution:3273` | None significant |
|
||
| **L2 retrieval** | on-device embed, paraphrase replay | **Live code, model-unverified** | `:1673`; `best_l2_match` tested `:5051` | Real-model separation only in an ignored test |
|
||
| **L3 on-device model** | GGUF/Mellum generates edits | **Live wiring, model-unverified** | `:1883`; `l3_generate:3540` | Off by default; edit-quality manual only |
|
||
| **L4 ACP federation** | Claude/Gemini/Codex | **Live** | `send_to_l4:2462` real `thread.send` | Receipt energy estimated, not measured |
|
||
| **Energy metering** | real counter, honest provenance | **Live (Apple ModelBased / Linux HwShunt), Estimator fallback** | `host_energy:3196`; baseline-subtracted | L4 estimator-only; Apple ModelBased not a shunt (tagged) |
|
||
| **JCR-1 sealing** | COSE_Sign1 ledger, chain | **Live** | `seal_op_receipt:738`; tamper test `:5252` | Op `author` zeroed; some seal failures `let _ =` |
|
||
| **JCP grant gating** | tiers structurally un/reachable | **Live** (strongest part) | `CascadeGate` real `JcpGate`; `authorize_tier:2611` | None |
|
||
| **Agent tool-gate** | read-only loop can't run/write | **Live** | `tool_gate.rs` real `LocalToolGate` | Ungated default still uses `is_allowed_command` prefix-match |
|
||
| **Sessions (worktrees)** | git worktree per session, gated merge | **Live (real git)** | `sessions.rs:404` `worktree add`, `:487` `git merge` | None significant |
|
||
| **Hooks PreTool/PreMerge** | deny-capable, fired | **Live & fired** | `authorize_tier:2623`; `merge_session:2855` | PostTool/SessionEnd never fired |
|
||
| **Control protocol** | newline-JSON Unix/TCP, peers/relay | **Live & wired** | `control.rs`; `main.rs:221` | Mutating cmds ack-then-async (disclosed) |
|
||
| **MCP bridge** | `--jc-mcp` stdio server | **Live & wired** | `main.rs:245` | Outbound MCP client tested only by ignored test |
|
||
| **Codebase index** | on-device embed + cosine | **Live code, model-unverified** | `codebase_index.rs` | Retrieval-quality only in ignored test |
|
||
| **FIM prediction** | Cursor-Tab on-device | **Live & registered** | `edit_prediction_local.rs`; registered `:115` | Infill quality unverified in CI |
|
||
| **Sandbox sealing** | S0–S7 attested receipt | **Live client, external-binary-dependent** | `sandbox_bridge.rs:67` | Needs `sandbox-mcp`; absent → silent skip |
|
||
| **Methodology/contracts** | sign spec, cite ops, frame_check | **Live (structural only)** | `methodology.rs` real ed25519 | Pre/post/inv inert, no proof discharge (disclosed) |
|
||
| **Conformance C1–C5** | vector pack passes in-tree | **Live** | `tests/conformance.rs`; non-vacuous guard | Pinned to one rev |
|
||
|
||
**Net:** the enforcement-and-accounting machinery is real and CI-tested; the weakness is concentrated in the loaded-model layer (ignored tests + manual runs) and the anonymized op author.
|