Follow-up to c2ab0bc (which zero-copied AppleAmx). Add
TensorView::f32_cow — borrowed &[f32] when F32+aligned (the
mmap'd/resident case), owned copy otherwise — and sweep every
reference-backend kernel's read-only input from as_f32_vec to
f32_cow (matmul, lookup, norm, rope, softmax, elementwise,
activation, conv1d, sample, quantized matmuls). The worst was
the embedding lookup copying the whole token-embedding table
element-by-element every step.
Rigorous note on the GGUF decode speed: profiling (kernel-sum vs
wall per step) shows time is ~99% in kernels, ~1% framework. The
catastrophic averages I measured earlier (~0.5 tok/s) were
page-cache THRASHING on a box loaded with concurrent builds
re-faulting the mmap'd weights each step — not a code defect.
Warm steady-state for tinyllama-1.1b-q8 is ~70 ms/step (~14
tok/s). The residual ~5x gap to llama.cpp is the known
architecture difference (f32-dequantized weights + AMX/CPU vs
quantized-in-kernel GEMV + Metal GPU), not a bug.
Tests: backend-reference 13, green. Output byte-coherent.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
a3432dd949