68ef34e17c* loader-gguf: dense Granite GGUF-file decoder (first GGUF-file forward)
Add `granite` - a pure-Rust dense-Granite (general.architecture="granite")
text forward that reads a real GGUF weight file (read_gguf_file mmap +
tensor_from_gguf Q8_0/Q4_K/Q5_K -> f32). This is the substrate's first decoder
that consumes a GGUF *file* end-to-end; every other validated forward (gemma4,
granite_h, mellum, cohere2_moe, nemotron_h) binds from a safetensors directory.
Granite's dense variant is Llama-shaped (separate q/k/v, RMSNorm, SwiGLU, GQA,
RoPE) plus four scalar multipliers that materially change the numerics and are
applied per llama.cpp's build_granite: embedding_scale, attention.scale (the
softmax scale, replacing 1/sqrt(head_dim)), residual_scale, logit_scale. Reuses
gemma4's linear/rmsnorm helpers.
Add BpeTokenizer::from_gguf: reconstruct the gpt2/dbrx byte-level tokenizer from
the GGUF's embedded tokenizer.ggml.{tokens,merges,token_type} and reuse the
validated from_json_bytes pipeline (BPE + ByteLevel pre-tokenizer).
Oracle-gated against llama.cpp on Granite-4.1-8B-Q8_0 (tests/real_granite_oracle,
ignored - needs the checkpoint): raw prompt "The capital of France is"
([791,6864,315,9822,374]) -> argmax 12366 (" Paris"); encode matches
llama-tokenize; greedy decode matches llama-simple. 281 lib tests green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
* loader-gguf: LFM2-MoE GGUF-file decoder (2nd GGUF forward, first hybrid)
Add `lfm2moe` - a pure-Rust LFM2.5 (general.architecture="lfm2moe") text
forward read directly from a GGUF file. Second GGUF-*file* decoder after
`granite`, and the first *hybrid* one: 24 blocks each `h += mixer(norm(h))`
then `h += mlp(norm(h))`, where per layer
- mixer = grouped-query attention with per-head q/k RMSNorm + RoPE (6 layers),
or a gated short convolution (in_proj -> split(B,C,x) -> B·x -> causal
depthwise conv1d(L=3) -> C·· -> out_proj) for the rest, and
- mlp = leading dense SwiGLU (first `leading_dense` blocks) or sigmoid-gated
top-k MoE (ffn_gate_inp router + exp_probs_b selection bias, 4-of-32
experts, weights = renormalised selected sigmoid probs).
Config, layer-type map (attn iff blk.N.attn_q present), and tokenizer all read
from the GGUF's own metadata; weights dequantised via tensor_from_gguf
(Q4_K / Q6_K / F32). Tied lm_head; final norm = token_embd_norm.
Oracle-gated vs llama.cpp (tests/real_lfm2_oracle, ignored - needs the ~5 GB
Q4_K_M checkpoint). NOTE: validate on a *sharp* prompt - "The quick brown fox
jumps over the lazy" (+BOS 124894) greedily -> " dog" (4448), matching
llama-simple. A flat prompt ("The capital of France is") is a poor oracle: the
f32 forward vs llama.cpp's Q4_K matmul can flip the argmax when the top logits
sit within ~0.1 of each other. 281 lib tests green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
* loader-gguf: Nemotron-H-MoE 30B GGUF decoder (lazy per-expert dequant)
Add `nemotron_gguf` - a pure-Rust Nemotron-H-MoE (arch "nemotron_h_moe") text
forward read directly from a GGUF file. Third GGUF-*file* decoder (after granite
dense + lfm2moe hybrid) and the first **30B-class MoE**: the 128 routed experts
stay quantized (mmap'd) and only the top-k selected per token are dequantised on
the fly (their Q5_0/Q8_0 blocks are per-expert contiguous), so 30B fits in
128 GB instead of the ~120 GB an all-f32 dequant would need. Backbone (~2.6B:
embeddings, mamba, attention, norms, router, shared expert) is f32.
Each of the 52 blocks is a single op (h += op(rmsnorm(h))), typed from the GGUF
per-layer arrays (head_count_kv[l]>0 → attention, feed_forward_length[l]>0 → MoE,
else Mamba):
- Mamba2 mixer: in_proj → conv1d(k=4) → SiLU → SSD scan → gated RMSNorm(z) →
out_proj, reusing crate::ssm::{ConvState, SsmState, mamba2_ssd_step,
gated_rmsnorm_inplace} (the recurrent state is threaded token-by-token).
- GQA attention (32q/2kv, head_dim 128, PARTIAL RoPE over rope.dimension_count,
no QK-norm) with a growing KV cache.
- Squared-ReLU MoE: sigmoid-gated top-6-of-128 (exp_probs_b selection bias,
weights renormalised then × expert_weights_scale), experts = down(relu²(up·x))
with NO gate proj, plus an always-on shared expert.
Also wire GGML type 6 (Q5_0) through tensor_from_gguf - the existing
dequant::dequantize_q5_0 was implemented but never dispatched; this GGUF uses
Q5_0 for token_embd / ssm_in / attn_q / ffn_up_exps.
Oracle-gated vs llama.cpp (tests/real_nemotron_oracle, ignored - needs the
~17 GB checkpoint): sharp prompt "The quick brown fox jumps over the lazy"
(no BOS) → argmax 10575 (" dog"). 281 lib tests green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
* loader-gguf: Qwen3.6/Qwen3-Next GatedDeltaNet GGUF decoder
Add `qwen35moe` - a pure-Rust Qwen3.6 (arch "qwen35moe" = Qwen3-Next) text
forward read directly from a GGUF file. Fourth GGUF-file decoder and the most
intricate: a **Gated-DeltaNet (linear attention) + gated full-attention hybrid
MoE**, ported op-for-op from llama.cpp src/models/qwen35moe.cpp +
delta-net-base.cpp (autoregressive path) and oracle-gated against it.
40 blocks, h += mixer(rmsnorm(h,attn_norm)) then h += moe(rmsnorm(h,post_attn)):
- Gated DeltaNet (layers (l+1)%full_attn_interval != 0): conv1d(in_proj(x))
→ SiLU, split q/k/v, L2-norm q/k, repeat k-heads→v-heads, then the delta
recurrence per v-head (128×128 state S):
S = S·exp(g); sk = Sᵀk; d = β(v−sk); S += k⊗d; o = Sᵀ(q/√dₖ)
g = softplus(alpha+dt_bias)·ssm_a, β = sigmoid(ssm_beta·x); then gated
RMSNorm rmsnorm(o,ssm_norm)·SiLU(z) and out_proj.
- Gated full attention (every full_attn_interval-th): fused q|gate projection,
per-head Q/K RMSNorm, partial RoPE (n_rot of head_dim; M-RoPE ≡ standard for
text), GQA softmax, then attn·sigmoid(gate) and out_proj.
- MoE: softmax top-k over 256 experts (SwiGLU, renormalised weights) + a
self-sigmoid-gated shared expert.
The 256 routed experts stay quantized (mmap'd); only the top-k selected per
token are dequantised (lazy per-expert, like nemotron) so the 35B-A3B fits RAM.
vocab is derived from the token_embd shape.
Oracle-gated vs llama.cpp (tests/real_qwen35_oracle, ignored - needs ~22 GB
checkpoint): "The quick brown fox jumps over the lazy" (no BOS) → argmax 5388
(" dog"). 281 lib tests green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
---------
Co-authored-by: Transaction Science <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
71 lines
2.9 KiB
Rust
71 lines
2.9 KiB
Rust
//! End-to-end LFM2-MoE (`lfm2moe` arch) GGUF decode oracle. Ignored by default —
|
|
//! needs the real ~5 GB Q4_K_M checkpoint on disk.
|
|
//!
|
|
//! Reference is `llama.cpp` on the same file. NOTE: LFM2 prepends a BOS
|
|
//! (`<|startoftext|>` = 124894). We validate on a *sharp* idiom prompt where the
|
|
//! next token is unambiguous, so the argmax cannot be flipped by the small
|
|
//! numerical difference between this f32 forward and llama.cpp's Q4_K matmul
|
|
//! (a flat prompt like "The capital of France is" is a poor oracle for that
|
|
//! reason — LFM2 there is genuinely torn between " city"/" Paris"/" the").
|
|
//!
|
|
//! Run: `cargo test -p jouleclaw-loader-gguf --release --features rayon \
|
|
//! --test real_lfm2_oracle -- --ignored --nocapture`
|
|
|
|
use jouleclaw_loader_gguf::bpe_tokenizer::BpeTokenizer;
|
|
use jouleclaw_loader_gguf::lfm2moe::Lfm2Moe;
|
|
use jouleclaw_loader_gguf::read_gguf_file;
|
|
|
|
const GG: &str =
|
|
"/Volumes/macos_4TB_external_1/neuronexus_ai/model_library/lfm2.5-8b-a1b-gguf/LFM2.5-8B-A1B-Q4_K_M.gguf";
|
|
|
|
const BOS: u32 = 124894;
|
|
/// "The quick brown fox jumps over the lazy" (+BOS) per `llama-tokenize`.
|
|
const FOX: [u32; 9] = [BOS, 597, 4104, 10299, 35886, 56085, 890, 278, 45698];
|
|
/// " dog" — llama.cpp's confident greedy next token.
|
|
const DOG: u32 = 4448;
|
|
|
|
fn argmax(v: &[f32]) -> usize {
|
|
v.iter()
|
|
.enumerate()
|
|
.fold((0usize, f32::NEG_INFINITY), |(bi, bv), (i, &x)| if x > bv { (i, x) } else { (bi, bv) })
|
|
.0
|
|
}
|
|
|
|
#[test]
|
|
#[ignore]
|
|
fn real_lfm2_predicts_dog() {
|
|
if !std::path::Path::new(GG).exists() {
|
|
eprintln!("skip: lfm2 GGUF not present");
|
|
return;
|
|
}
|
|
let m = Lfm2Moe::load(GG).expect("load lfm2 gguf");
|
|
eprintln!(
|
|
"lfm2moe: {} layers, d={}, heads={}/{}, experts={}/{}, dense_lead={}, L={}, vocab={}",
|
|
m.cfg.n_layers, m.cfg.d_model, m.cfg.n_heads, m.cfg.n_kv, m.cfg.n_used, m.cfg.n_experts,
|
|
m.cfg.leading_dense, m.cfg.l_cache, m.cfg.vocab
|
|
);
|
|
let got = argmax(&m.forward(&FOX));
|
|
eprintln!("argmax = {got} (expected {DOG} = ' dog')");
|
|
assert_eq!(got, DOG as usize, "greedy next token diverges from llama.cpp");
|
|
}
|
|
|
|
/// Full text-in / text-out through the GGUF-embedded tokenizer (+BOS): the prompt
|
|
/// tokenizes to the llama.cpp ids and greedy decode reads "... dog".
|
|
#[test]
|
|
#[ignore]
|
|
fn real_lfm2_text_roundtrip() {
|
|
if !std::path::Path::new(GG).exists() {
|
|
eprintln!("skip: lfm2 GGUF not present");
|
|
return;
|
|
}
|
|
let tok = BpeTokenizer::from_gguf(&read_gguf_file(GG).expect("gguf")).expect("tokenizer");
|
|
let body = tok.encode("The quick brown fox jumps over the lazy");
|
|
assert_eq!(body, FOX[1..].to_vec(), "tokenization diverges from llama.cpp");
|
|
let mut ids = vec![BOS];
|
|
ids.extend_from_slice(&body);
|
|
|
|
let m = Lfm2Moe::load(GG).expect("load");
|
|
let text = tok.decode(&m.generate(&ids, 3));
|
|
eprintln!("generated = {text:?}");
|
|
assert!(text.starts_with(" dog"), "greedy decode should start ' dog', got {text:?}");
|
|
}
|