joule_core: LFM2 on-device, semantic code graph, and codebase-health analysis #1
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "joule-semantic-graph-fusion-forge"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Six commits across three threads of work. All deterministic/on-device; no cloud dependency added.
1. Semantic code graph + embedding-fused auto-context
code_graph— a queryable symbol/reference map (where isXdefined, who calls it): the LogicLens-style navigation that ranking (repo_map) and retrieval don't provide. Control method:symbol.run_agent_loopnow takes aretrieve_semanticclosure; the GUI embeds the intent and fuses cosine-nearest chunks with the lexical + structural hits. The daemon passes a no-op and degrades cleanly.2. Run Liquid AI LFM2 on-device
generate()decodes via a step-graph builder that assumes every block is attention, so LFM2 — which pairs gated short-convolution blocks with a few GQA blocks — died on the first conv layer huntingblk.0.attn_q.weight. Routed the L3 GGUF backend throughConversationinstead, which builds the per-layer-aware graph (wiring the recurrent conv state) and caches the compiled step across tokens — so LFM2 runs and every arch decodes faster.Also made on-device instruct models usable for structured output: prompts are wrapped in the model's declared chat template (detected from GGUF metadata) and stopped at the turn-end marker the configured
eosoften omits (FIM stays raw);l3_generate_budgetedgives a decomposition plan room to finish as one JSON object;parse_planerrors now carry the model's actual output.Verified: LFM2-1.2B/2.6B load, generate coherent text, and run the full agent path (~12–18 tok/s release).
3. Codebase-health analysis (CodeFlow port)
New
joule_core::codehealth— a Rust port of the deterministic analysis core of CodeFlow (MIT), rebuilt oncode_graphrather than re-parsing: every symbol's definers + call sites already are the file→file dependency graph these metrics run over. Energy-free (graph walks + string scans, no model, no per-language parser).blast_radius— "if I change this file, what breaks?" (direct + depth-decayed transitive dependents)circular_dependencies— strongly-connected components (iterative Tarjan; naive path enumeration was exponential and timed out on the real graph)coupling,security_scan,health(A–F grade)clone_groups— structural fingerprint + LCS-confirmed duplicate detectioncyclomatic_complexity— ranked hotspotssuggestions— prioritized, actionable recommendations synthesized from the findingsControl methods:
health,clones,complexity.Precision work, because a tool that cries wolf gets ignored: only single-definer symbols create dependency edges (generic names like
new/runotherwise wire every file to every other — cut joule_core's edges 2553→1118); the secret rule requires the secret word to be the assigned identifier; dead-code excludes public API (apubsymbol's callers live outside the tree — 608→359); and test detection reads content, since Rust keeps tests inline in#[cfg(test)]and a path-only check reported 0% coverage for a fully-tested crate.Found real duplication on first run — in CodeFlow's own
card/lib(receipt-md.jsvsreceipt.js, 100%) and injoule_coreitself (sandbox_bridgevsmcp_client, 98%).Verification
joule_coresuite green (176 passed), daemon + GUI panel build clean.Release Notes:
Add code_graph, a queryable symbol/reference map over a repo: where a symbol is defined and who calls it. This is the LogicLens-style repo understanding the July SOTA scan flagged as the last open gap -- it complements repo_map (ranks files by PageRank) and retrieval (fuses search signals) by answering precise navigation questions neither does. Built from the same plain-tokenizer symbol extraction repo_map already uses (no per-language parser -- best-effort, deterministic, energy-free). Exposed over the control protocol as a `symbol` method: `{"symbol":"name"}` returns its definition + call sites; no symbol returns the graph's symbol count. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>Add joule_core::codehealth — a deterministic, energy-free codebase-health layer built on the existing symbol graph. A Rust port of the analysis core of CodeFlow (github.com/braedonsaunders/codeflow, MIT), rebuilt on code_graph rather than re-parsing: every symbol's definers + call sites already ARE the file->file dependency graph these metrics run over. - blast_radius: "if I change this file, what breaks?" — direct + depth-decayed transitive dependents (reverse reachability), the flagship query. - circular_dependencies: strongly-connected components (iterative Tarjan, linear) — an SCC of >=2 files is a dependency cycle. - coupling: fan-in/fan-out, average, and the high-fan-in hubs. - security_scan: string rules (no regex dep) for hardcoded secrets, string-built SQL, eval/dynamic exec, shell exec, and left-in debug logging. - health: the CodeFlow 100-minus-capped-penalties formula -> an A-F grade. - analyze(): whole-repo report; exposed over the control protocol as `health` ({"file":"path"} returns just that file's blast radius). Precision: only symbols with a SINGLE definer produce edges — the token-based graph can't attribute a name defined in many files (`new`, `run`, `build`) to one place, and counting those wires every file to every other. The secret rule requires the secret word to be the assigned identifier (not any mention) with a long literal, so a codebase that legitimately deals in tokens/auth isn't drowned in false positives. Verified on two real repos: joule_core (36 files) and CodeFlow's own card/lib (27 files, coupling avg 5.1). 5 unit tests; full joule_core suite green (171). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>Extend joule_core::codehealth with the rest of CodeFlow's quality analysis: - code_fingerprint: structural signature (identifiers/strings/numbers erased, summarized as L<loops>C<conds>F<calls>R<returns>S<len>) for grouping clones. - code_similarity / lcs_length: normalized longest-common-subsequence ratio to confirm a fingerprint match is a real clone (capped at 500 chars, cheap). - clone_groups: extract function-sized brace-balanced blocks across the repo, group by fingerprint, confirm each group by minimum pairwise similarity (>=85%). - cyclomatic_complexity / complexity_level: 1 + control-flow branch points, bucketed low/medium/high/critical. Exposed over the control protocol as `clones` and `complexity` (the latter ranks per-file hotspots, or scores one file with {"file":"path"}). Found real duplication on first run: in CodeFlow's own card/lib (receipt-md.js vs receipt.js, 100%) and in joule_core itself (sandbox_bridge vs mcp_client 98%, an intra-file dup in sessions.rs 92%). 8 codehealth tests; suite green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>joule_core: semantic code graph and embedding-fused auto-contextto joule_core: LFM2 on-device, semantic code graph, and codebase-health analysisIngest qmd's `hyde` sub-query: instead of embedding the bare query, have the on-device model write the code that would answer it and embed THAT. A question ("how is the cache invalidated?") and its implementation share few literal tokens; a hypothetical answer is phrased in the code's own vocabulary, so its embedding lands nearer the real implementation. - retrieval::hyde_prompt(query): the model-free prompt (the caller runs the generation + embed, so joule_core stays model-free). - The GUI's auto-context semantic closure uses it when JOULE_CODE_HYDE is set: generate a short hypothetical, embed that, fuse as the semantic signal. Off by default (it costs one generation); falls back to the raw query on empty/error. Rounds out the qmd retrieval ingestion (BM25 lexical + RRF top-rank bonus + HyDE) against the RRF fusion + L2 index agent-joule already had. 10 retrieval tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>Toolchain and dependencies: - rust-toolchain.toml 1.96.0 -> 1.97.0 (current stable). - cargo update across the workspace: latest semver-compatible versions. - All open-standards git deps (jouleclaw-*, joule-code-*, sandbox-*, joule-moq) move together 320fa1e -> c4b536cf, and off github.com onto the sovereign forge (git.transaction.science) — the flagged org is 404 to anonymous clients, which breaks clean-container CI even though a warm dev cache hides it. c4b536cf is 320fa1e plus the additive score_binary commit, deliberately NOT the newer fix/omni-build branch: that line has dropped crates this workspace still depends on (jcp-a2a), so bumping onto it would mean reconciling against a restructured open-standards rather than taking a runtime addition. Cross-encoder reranking (the last stage of qmd's pipeline): - l3::l3_rerank_score — relevance of a document to a query in [0,1], read as P(yes) after ONE forward pass over a yes/no reranker prompt. Resolves the yes/no ids by exact token lookup and encodes the chat markers atomically; the prompt omits the reasoning <think> block, which derails the 0.6B reranker. - retrieval::rerank — reorders fused hits by an injected scorer (model-free here); an unscored hit keeps its fused rank rather than mis-ordering. - `search` control method: fuse BM25 + structural, then rerank the top with the loaded model ({"rerank": false} to skip). Fusion alone ranks cascade.rs first for "invalidate the embedding cache"; with Qwen3-Reranker-0.6B the model re-reads each candidate against the query and demotes fabric.rs (compute fabric, not caches) from 3rd to last. 182 joule_core tests green on 1.97; joule_coded builds release. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.