68ef34e17c* loader-gguf: dense Granite GGUF-file decoder (first GGUF-file forward)
Add `granite` - a pure-Rust dense-Granite (general.architecture="granite")
text forward that reads a real GGUF weight file (read_gguf_file mmap +
tensor_from_gguf Q8_0/Q4_K/Q5_K -> f32). This is the substrate's first decoder
that consumes a GGUF *file* end-to-end; every other validated forward (gemma4,
granite_h, mellum, cohere2_moe, nemotron_h) binds from a safetensors directory.
Granite's dense variant is Llama-shaped (separate q/k/v, RMSNorm, SwiGLU, GQA,
RoPE) plus four scalar multipliers that materially change the numerics and are
applied per llama.cpp's build_granite: embedding_scale, attention.scale (the
softmax scale, replacing 1/sqrt(head_dim)), residual_scale, logit_scale. Reuses
gemma4's linear/rmsnorm helpers.
Add BpeTokenizer::from_gguf: reconstruct the gpt2/dbrx byte-level tokenizer from
the GGUF's embedded tokenizer.ggml.{tokens,merges,token_type} and reuse the
validated from_json_bytes pipeline (BPE + ByteLevel pre-tokenizer).
Oracle-gated against llama.cpp on Granite-4.1-8B-Q8_0 (tests/real_granite_oracle,
ignored - needs the checkpoint): raw prompt "The capital of France is"
([791,6864,315,9822,374]) -> argmax 12366 (" Paris"); encode matches
llama-tokenize; greedy decode matches llama-simple. 281 lib tests green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
* loader-gguf: LFM2-MoE GGUF-file decoder (2nd GGUF forward, first hybrid)
Add `lfm2moe` - a pure-Rust LFM2.5 (general.architecture="lfm2moe") text
forward read directly from a GGUF file. Second GGUF-*file* decoder after
`granite`, and the first *hybrid* one: 24 blocks each `h += mixer(norm(h))`
then `h += mlp(norm(h))`, where per layer
- mixer = grouped-query attention with per-head q/k RMSNorm + RoPE (6 layers),
or a gated short convolution (in_proj -> split(B,C,x) -> B·x -> causal
depthwise conv1d(L=3) -> C·· -> out_proj) for the rest, and
- mlp = leading dense SwiGLU (first `leading_dense` blocks) or sigmoid-gated
top-k MoE (ffn_gate_inp router + exp_probs_b selection bias, 4-of-32
experts, weights = renormalised selected sigmoid probs).
Config, layer-type map (attn iff blk.N.attn_q present), and tokenizer all read
from the GGUF's own metadata; weights dequantised via tensor_from_gguf
(Q4_K / Q6_K / F32). Tied lm_head; final norm = token_embd_norm.
Oracle-gated vs llama.cpp (tests/real_lfm2_oracle, ignored - needs the ~5 GB
Q4_K_M checkpoint). NOTE: validate on a *sharp* prompt - "The quick brown fox
jumps over the lazy" (+BOS 124894) greedily -> " dog" (4448), matching
llama-simple. A flat prompt ("The capital of France is") is a poor oracle: the
f32 forward vs llama.cpp's Q4_K matmul can flip the argmax when the top logits
sit within ~0.1 of each other. 281 lib tests green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
* loader-gguf: Nemotron-H-MoE 30B GGUF decoder (lazy per-expert dequant)
Add `nemotron_gguf` - a pure-Rust Nemotron-H-MoE (arch "nemotron_h_moe") text
forward read directly from a GGUF file. Third GGUF-*file* decoder (after granite
dense + lfm2moe hybrid) and the first **30B-class MoE**: the 128 routed experts
stay quantized (mmap'd) and only the top-k selected per token are dequantised on
the fly (their Q5_0/Q8_0 blocks are per-expert contiguous), so 30B fits in
128 GB instead of the ~120 GB an all-f32 dequant would need. Backbone (~2.6B:
embeddings, mamba, attention, norms, router, shared expert) is f32.
Each of the 52 blocks is a single op (h += op(rmsnorm(h))), typed from the GGUF
per-layer arrays (head_count_kv[l]>0 → attention, feed_forward_length[l]>0 → MoE,
else Mamba):
- Mamba2 mixer: in_proj → conv1d(k=4) → SiLU → SSD scan → gated RMSNorm(z) →
out_proj, reusing crate::ssm::{ConvState, SsmState, mamba2_ssd_step,
gated_rmsnorm_inplace} (the recurrent state is threaded token-by-token).
- GQA attention (32q/2kv, head_dim 128, PARTIAL RoPE over rope.dimension_count,
no QK-norm) with a growing KV cache.
- Squared-ReLU MoE: sigmoid-gated top-6-of-128 (exp_probs_b selection bias,
weights renormalised then × expert_weights_scale), experts = down(relu²(up·x))
with NO gate proj, plus an always-on shared expert.
Also wire GGML type 6 (Q5_0) through tensor_from_gguf - the existing
dequant::dequantize_q5_0 was implemented but never dispatched; this GGUF uses
Q5_0 for token_embd / ssm_in / attn_q / ffn_up_exps.
Oracle-gated vs llama.cpp (tests/real_nemotron_oracle, ignored - needs the
~17 GB checkpoint): sharp prompt "The quick brown fox jumps over the lazy"
(no BOS) → argmax 10575 (" dog"). 281 lib tests green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
* loader-gguf: Qwen3.6/Qwen3-Next GatedDeltaNet GGUF decoder
Add `qwen35moe` - a pure-Rust Qwen3.6 (arch "qwen35moe" = Qwen3-Next) text
forward read directly from a GGUF file. Fourth GGUF-file decoder and the most
intricate: a **Gated-DeltaNet (linear attention) + gated full-attention hybrid
MoE**, ported op-for-op from llama.cpp src/models/qwen35moe.cpp +
delta-net-base.cpp (autoregressive path) and oracle-gated against it.
40 blocks, h += mixer(rmsnorm(h,attn_norm)) then h += moe(rmsnorm(h,post_attn)):
- Gated DeltaNet (layers (l+1)%full_attn_interval != 0): conv1d(in_proj(x))
→ SiLU, split q/k/v, L2-norm q/k, repeat k-heads→v-heads, then the delta
recurrence per v-head (128×128 state S):
S = S·exp(g); sk = Sᵀk; d = β(v−sk); S += k⊗d; o = Sᵀ(q/√dₖ)
g = softplus(alpha+dt_bias)·ssm_a, β = sigmoid(ssm_beta·x); then gated
RMSNorm rmsnorm(o,ssm_norm)·SiLU(z) and out_proj.
- Gated full attention (every full_attn_interval-th): fused q|gate projection,
per-head Q/K RMSNorm, partial RoPE (n_rot of head_dim; M-RoPE ≡ standard for
text), GQA softmax, then attn·sigmoid(gate) and out_proj.
- MoE: softmax top-k over 256 experts (SwiGLU, renormalised weights) + a
self-sigmoid-gated shared expert.
The 256 routed experts stay quantized (mmap'd); only the top-k selected per
token are dequantised (lazy per-expert, like nemotron) so the 35B-A3B fits RAM.
vocab is derived from the token_embd shape.
Oracle-gated vs llama.cpp (tests/real_qwen35_oracle, ignored - needs ~22 GB
checkpoint): "The quick brown fox jumps over the lazy" (no BOS) → argmax 5388
(" dog"). 281 lib tests green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
---------
Co-authored-by: Transaction Science <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
39 lines
1.7 KiB
Rust
39 lines
1.7 KiB
Rust
//! End-to-end Qwen3.6 / Qwen3-Next (`qwen35moe`) GGUF decode oracle. Ignored by
|
|
//! default — needs the ~22 GB Q5_K_M 35B-A3B checkpoint. Validates the
|
|
//! Gated-DeltaNet + gated-attention hybrid MoE path (lazy per-expert dequant) on
|
|
//! a *sharp* prompt.
|
|
//!
|
|
//! Reference (llama.cpp / llama-simple, no BOS): "The quick brown fox jumps over
|
|
//! the lazy" -> " dog" (5388).
|
|
//!
|
|
//! Run: `cargo test -p jouleclaw-loader-gguf --release --features rayon \
|
|
//! --test real_qwen35_oracle -- --ignored --nocapture`
|
|
|
|
use jouleclaw_loader_gguf::qwen35moe::Qwen35Moe;
|
|
|
|
const GG: &str =
|
|
"/Volumes/macos_4TB_external_1/neuronexus_ai/model_library/qwen3.6-35b-a3b-gguf/Qwen3.6-35B-A3B-UD-Q5_K_M.gguf";
|
|
|
|
const FOX: [u32; 8] = [760, 3841, 13477, 37550, 33075, 888, 279, 15217];
|
|
const DOG: u32 = 5388;
|
|
|
|
#[test]
|
|
#[ignore]
|
|
fn real_qwen35_predicts_dog() {
|
|
if !std::path::Path::new(GG).exists() {
|
|
eprintln!("skip: qwen35moe GGUF not present");
|
|
return;
|
|
}
|
|
let m = Qwen35Moe::load(GG).expect("load qwen35moe gguf");
|
|
eprintln!(
|
|
"qwen35moe: {} layers (fa every {}), d={}, attn {}q/{}kv hd={} nrot={}, deltanet {}k/{}v heads hd={}, experts {}/{} +shared",
|
|
m.cfg.n_layers, m.cfg.full_attn_interval, m.cfg.d_model, m.cfg.n_head, m.cfg.n_kv,
|
|
m.cfg.head_dim, m.cfg.n_rot, m.cfg.n_k_heads, m.cfg.n_v_heads, m.cfg.head_v,
|
|
m.cfg.n_used, m.cfg.n_experts
|
|
);
|
|
let logits = m.forward(&FOX);
|
|
let mut idx: Vec<usize> = (0..logits.len()).collect();
|
|
idx.sort_by(|&a, &b| logits[b].partial_cmp(&logits[a]).unwrap());
|
|
eprintln!("top5 = {:?} (expected {DOG} = ' dog')", &idx[..5]);
|
|
assert_eq!(idx[0], DOG as usize, "greedy next token diverges from llama.cpp");
|
|
}
|