jouleclaw: North decode 3-6x and prefill 6-11x on the CPU, bit-identical #649

Merged

dcharlot merged 17 commits from jouleclaw/north-decode-speed into main 2026-11-07 03:25:35 -05:00

Conversation 0 Commits 17 Files changed 10 +3284 -58

dcharlot commented 2026-11-05 10:19:21 -05:00

Owner

Copy link

North Mini Code 1.0 (Cohere2-MoE, 30B / 3B active, BF16) is the model WAI Quantum's program-writing loop runs on. Its pure-Rust CPU path (--features rayon) now decodes 3–6× and prefills 6–11× faster, with bit-identical logits.

Profile (base, JOULECLAW_PROF=1, M3 Ultra under load, ~600-token prompt)

• decode, 212 ms/token: attention 26% (all 32 heads on one thread, one scalar add chain per score), q/k/v 20%, o_proj 20%, experts 28%, lm_head 3%. lm_head streamed 1 GB in ~7 ms (~140 GB/s), while each layer's 16–50 MB projections moved 12–40 GB/s. The time went to ~7 dispatches per layer, three of them single-threaded GEMVs, and to a GEMV kernel held to ~16–20 GB/s per core by its two FMA chains. Memory bandwidth was not the limit.
• prefill, 13 tok/s: attention 36% on one thread, experts 50% run one at a time.

Changes (each output keeps its exact reduction order)

1. BF16 kernel: register blocks of up to 4 activation rows × 4 weight rows (16 chains instead of 2), a 1 KiB prefetch, 64 KiB L1 tiles. Threaded GEMVs are split into ~256 KiB chunks; prefill GEMMs into 64 × 64 tiles. The old loop stays as matmul_bf16_neon_one_row and is tested against the new one bit for bit.

2. attention_row_f32: attention_forward_f32's exact arithmetic, with 16 keys' scores in NEON lanes (4×4 transposes) and V running sums held in registers.

3. North CPU passes:

   • decode in three dispatches per layer, on spin_pool, whose threads stay awake between dispatches;
   • prefill attention per (head, 16 rows), experts in parallel;
   • RoPE angles once per position;
   • forward_extend, which batches a follow-up turn into an existing context;
   • one thread per performance core, at most 16 (JOULECLAW_THREADS overrides).

   The old passes stay as forward_*_reference, which is the fabric path.

Token identity

• Real checkpoint: five prompts (a raw completion, Python, OpenQASM, 596 and 2698 tokens of Rust) × 72 greedy tokens give the same token ids and the same hash of every logit's bits at every step as main. That holds on the M3 Ultra and on an M4 Max, and the two machines agree with each other. Every timing repeat produced the same hash sequence.
• forward_extend: equal to feeding the same 69 tokens one at a time.
• Unit tests pin, bit for bit:
  • the blocked kernel, the chunks and the tiles against the one-row loop;
  • the attention row against attention_forward_f32 (zero-weight underflow included);
  • the CPU prefill, decode and extend against the reference passes, across the sliding window.

Before / after

Medians, base and new interleaved in the same conditions; four runs each (three for base at 12 threads). The M3 Ultra was shared with another North benchmark and CI (load average 30–130); the M4 Max was lightly loaded.

	M3 Ultra base	M3 Ultra base, 12 thr	M3 Ultra new	M4 Max base	M4 Max new
decode at ~600 context	267 ms	168 ms	54 ms	116 ms	35 ms
decode at ~2700 context	405 ms	364 ms	80 ms	246 ms	48 ms
prefill, 596 tokens	9.9 tok/s	12.0 tok/s	91 tok/s	17.8 tok/s	105 tok/s
prefill, 2698 tokens	5.9 tok/s	4.5 tok/s	63 tok/s	9.0 tok/s	79 tok/s

Paired run-for-run ratios and all runs are in BENCHMARKS.md. The run-to-run spread on the shared host is large; the interleaved pairs are the fair reading.

Checks

The crate's tests are green on macOS arm64, natively on the M3 Ultra: 311 with rayon, 304 with no features, and the targeted set under rayon,fabric. cargo check is clean for aarch64- and x86_64-unknown-linux-gnu, with and without rayon. The public-names check is clean.

🤖 Generated with Claude Code

North Mini Code 1.0 (Cohere2-MoE, 30B / 3B active, BF16) is the model WAI Quantum's program-writing loop runs on. Its pure-Rust CPU path (`--features rayon`) now decodes 3–6× and prefills 6–11× faster, with **bit-identical logits**. ## Profile (base, `JOULECLAW_PROF=1`, M3 Ultra under load, ~600-token prompt) - **decode, 212 ms/token:** attention 26% (all 32 heads on one thread, one scalar add chain per score), q/k/v 20%, o_proj 20%, experts 28%, lm_head 3%. lm_head streamed 1 GB in ~7 ms (~140 GB/s), while each layer's 16–50 MB projections moved 12–40 GB/s. The time went to ~7 dispatches per layer, three of them single-threaded GEMVs, and to a GEMV kernel held to ~16–20 GB/s per core by its two FMA chains. Memory bandwidth was not the limit. - **prefill, 13 tok/s:** attention 36% on one thread, experts 50% run one at a time. ## Changes (each output keeps its exact reduction order) 1. **BF16 kernel:** register blocks of up to 4 activation rows × 4 weight rows (16 chains instead of 2), a 1 KiB prefetch, 64 KiB L1 tiles. Threaded GEMVs are split into ~256 KiB chunks; prefill GEMMs into 64 × 64 tiles. The old loop stays as `matmul_bf16_neon_one_row` and is tested against the new one bit for bit. 2. **`attention_row_f32`:** `attention_forward_f32`'s exact arithmetic, with 16 keys' scores in NEON lanes (4×4 transposes) and V running sums held in registers. 3. **North CPU passes:** - decode in three dispatches per layer, on `spin_pool`, whose threads stay awake between dispatches; - prefill attention per (head, 16 rows), experts in parallel; - RoPE angles once per position; - `forward_extend`, which batches a follow-up turn into an existing context; - one thread per performance core, at most 16 (`JOULECLAW_THREADS` overrides). The old passes stay as `forward_*_reference`, which is the fabric path. ## Token identity - **Real checkpoint:** five prompts (a raw completion, Python, OpenQASM, 596 and 2698 tokens of Rust) × 72 greedy tokens give the same token ids **and the same hash of every logit's bits at every step** as `main`. That holds on the M3 Ultra and on an M4 Max, and the two machines agree with each other. Every timing repeat produced the same hash sequence. - **`forward_extend`:** equal to feeding the same 69 tokens one at a time. - **Unit tests** pin, bit for bit: - the blocked kernel, the chunks and the tiles against the one-row loop; - the attention row against `attention_forward_f32` (zero-weight underflow included); - the CPU prefill, decode and extend against the reference passes, across the sliding window. ## Before / after Medians, base and new interleaved in the same conditions; four runs each (three for base at 12 threads). The M3 Ultra was shared with another North benchmark and CI (load average 30–130); the M4 Max was lightly loaded. | | M3 Ultra base | M3 Ultra base, 12 thr | M3 Ultra new | M4 Max base | M4 Max new | |---|---|---|---|---|---| | decode at ~600 context | 267 ms | 168 ms | **54 ms** | 116 ms | **35 ms** | | decode at ~2700 context | 405 ms | 364 ms | **80 ms** | 246 ms | **48 ms** | | prefill, 596 tokens | 9.9 tok/s | 12.0 tok/s | **91 tok/s** | 17.8 tok/s | **105 tok/s** | | prefill, 2698 tokens | 5.9 tok/s | 4.5 tok/s | **63 tok/s** | 9.0 tok/s | **79 tok/s** | Paired run-for-run ratios and all runs are in `BENCHMARKS.md`. The run-to-run spread on the shared host is large; the interleaved pairs are the fair reading. ## Checks The crate's tests are green on macOS arm64, natively on the M3 Ultra: 311 with rayon, 304 with no features, and the targeted set under rayon,fabric. `cargo check` is clean for aarch64- and x86_64-unknown-linux-gnu, with and without rayon. The public-names check is clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 4 commits 2026-11-05 10:19:21 -05:00

jouleclaw: a per-op profile and a bit-identity bench for North decode ... 6db018cd04

JOULECLAW_PROF=1 times every op of the Cohere2-MoE forward passes
(rmsnorm, q/k/v, rope and the KV append, attention, o_proj, router,
the expert stages, the residual, lm_head). It reads clocks only.

north_bench runs one prompt from a file and prints, per decoded
token, its id and a hash of every logit's bits, so two builds can be
compared bit for bit, with the median ms per token and the profile.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw: a blocked BF16 kernel that keeps every dot product's order ... 9d928f493e

matmul_bf16_neon computed one weight row at a time on two accumulator
chains, so a core waited on the fused multiply-add's latency: about
16-20 GB/s of weight per core, in cache or not. It now computes a
block of up to 4 activation rows x 4 weight rows at once (up to 16
chains, shared loads), walks weight rows in 64 KiB L1 tiles, and
prefetches each row 1 KiB ahead. Every output is still accumulated
exactly as the one-row loop does it (the same lanes, the same chunk
order, the same final reduction, tail and bias), so results do not
change by a bit; the old loop stays as matmul_bf16_neon_one_row and
the tests compare the two bit for bit over block shapes, tails and
biases.

One core, measured: 61 against 20 GB/s on cache-resident weights
(M4 Max), and from DRAM 26 against 19 (M4 Max) and 16-17 against
14-15 without the prefetch (M3 Ultra).

The threaded kernel splits an m = 1 GEMV into small row chunks
(about 256 KiB, at least four per thread) instead of one fixed share
per thread, so a slow or descheduled thread no longer holds up the
rest, and splits m > 1 into 64 x 64 output tiles written in place,
so a prefill reads each weight about m/64 times instead of once per
activation row. GemvChunk / push_gemv_chunks expose the chunks, and
cpu_pool / in_cpu_pool give the CPU passes a pool sized by
JOULECLAW_THREADS (else RAYON_NUM_THREADS, else one thread per
performance core, at most 16).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw: one attention row, bit-identical to attention_forward_f32, fast ... 3fed234fd2

attention_row_f32 computes one query row against keys 0..kv_len with
attention_forward_f32's exact arithmetic. The reference adds each
score's products on one scalar chain (an f32 sum may not be
reordered), so it waits on the add's latency for every product.
Here four keys share a NEON register (a 4x4 transpose of four key
rows) and four registers run at once, each lane still multiplying,
rounding, adding and rounding in d order; the weighted sum of V
keeps its running sums in registers across keys, multiply then add
in key order, skipping zero weights as the reference does. The test
compares it with the reference bit for bit for decode and causal
rows, head widths that are and are not multiples of 4, key counts
around the blocks, and weights that underflow to zero.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw: North decode 3-6x and prefill 6-11x on the CPU, bit-identical ... 624b8389df

The profile (JOULECLAW_PROF=1, the M3 Ultra build host under load,
a ~600-token prompt) showed North Mini Code's CPU path waiting, not
streaming: decode spent 212 ms/token in ~7 dispatches per layer,
three of them single-threaded GEMVs, with all 32 attention heads on
one thread; its 16-50 MB projections moved 12-40 GB/s while the
1 GB lm_head, one dispatch, moved ~140 GB/s. Prefill (13 tok/s)
spent 36% in attention on one thread and 50% running the experts
one after another. Base decode was also fastest at 8 threads and
slowest at the default 32.

Cohere2MoeModel's forward passes now have CPU versions, used when
nothing may leave the CPU (always without the fabric feature):

- decode: three dispatches per layer (q/k/v + router; attention
  heads + expert gate/up; o_proj + expert down) of ~256 KiB row
  chunks, on spin_pool, a pool whose threads spin briefly between
  dispatches instead of sleeping and claim units one at a time;
- prefill: attention per (head, 16 query rows) in parallel, the
  experts in parallel, projections as 64 x 64 tiles;
- RoPE angles once per position;
- forward_extend: batched feeding of a state that already holds a
  context, equal to one forward_token per token (the program loop's
  repair rounds feed their turns that way today);
- threads: one per performance core, at most 16
  (JOULECLAW_THREADS overrides).

Every output keeps its reduction order, so nothing changes by a
bit. The previous passes stay as forward_*_reference (the fabric
path), and tests compare the CPU passes with them bit for bit,
prefill, decode and extend, across the sliding window. On the real
checkpoint, five prompts (6 to 2698 tokens) x 72 greedy tokens
gave base's token ids and the same hash of every step's logits, on
the M3 Ultra and on an M4 Max.

Medians, base vs new, interleaved (M3 Ultra shared with another
North benchmark and CI, load 30-130 / M4 Max lightly loaded):
decode at ~600 context 267 -> 54 ms / 116 -> 35 ms; at ~2700
context 405 -> 80 ms / 246 -> 48 ms; prefill of 596 tokens
9.9 -> 91 / 17.8 -> 105 tok/s, of 2698 tokens 5.9 -> 63 /
9.0 -> 79 tok/s. BENCHMARKS.md has every run and the paired
ratios. north_bench gains --then / --then-seq to time and check
forward_extend.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 4 commits 2026-11-05 17:32:00 -05:00

Merge main into jouleclaw/north-decode-speed ... d4632532d5

Brings in the ten commits main gained since this branch left it. None of them touches jouleclaw/; the merge is clean.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

jouleclaw-loader-gguf: the two lints the macOS job denies ... a947e2ed12

The macOS energy job ends by linting this crate with warnings
denied, in two configurations. It failed in the first and so never
reached the second:

- attention.rs tested `d_head % 4 == 0` in two places
  (manual_is_multiple_of). They read `d_head.is_multiple_of(4)`
  now, as the rest of the crate does; the value is the same for
  every d_head.
- behind it, with `rayon` on, matmul_bf16_threaded's tile closure
  rebound `ptr` to itself (redundant_locals). The rebinding was
  not idle: it made the closure capture the OutPtr, which is
  Sync, where naming `ptr.0` alone captures the raw pointer,
  which is not. OutPtr::get does the same by being a method, so
  the closure calls that and the rebinding goes.

Neither changes a result: the same branch is taken and the same
pointer is written through. The tests that pin both paths bit for
bit pass on macOS arm64, Linux aarch64 and Linux x86_64:
attention_row_is_bit_identical_to_reference, over head widths
that are and are not multiples of four, and
bf16_threaded_above_the_floor_is_bit_identical, over the tiled
shapes.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

jouleclaw-loader-gguf: the CPU-against-reference tests say when the reference has left the CPU ... 0223103676

In a `fabric` build one of fabric_dispatch's tests sets
JOULECLAW_FABRIC=1, and that switch is one per process. Every
test that reads it afterwards runs with the fabric engaged, and
the reference passes then hand their projections to the fabric's
members, which round differently. cpu_passes_match_reference_medium
and extend_matches_sequential_decode_medium compared the CPU
passes with those and failed whenever the two ran in one process
(five runs of five on the M3 Ultra). Alone they pass, five of
five; with the switch exported they fail the same way. The tiny
ones passed only because their shapes are below the fabric's
floors.

No identity is claimed with the fabric engaged: forward_token and
forward_prompt take the reference passes then precisely because
those may leave the CPU. So both helpers ask first, say on stderr
that they are not checking, and return. Asking settles the switch
for the process, so a test that goes on runs with the fabric off
to its end. Without the `fabric` feature the answer is always
"on the CPU", so the default and `rayon` runs check everything
they did.

No CI job runs these tests in a `fabric` build. This was found
running the crate's lib tests under `rayon fabric-meter` by hand,
and that run is now 321 passed, 2 ignored.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

Merge main into jouleclaw/north-decode-speed (#653 has landed) ... 1feb927554

Main gained the marks change while this branch was being checked. It touches wai/ and CHARTER.md only; jouleclaw/ is the tree the checks ran on.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

dcharlot added 5 commits 2026-11-06 03:10:04 -05:00

jouleclaw-loader-gguf: a late thread of a finished loop claims nothing of the next, and a unit's panic leaves the pool usable ... 16c1a6f252

The spin pool published a new loop's unit count before it replaced the
claim word, which still read (the finished loop's epoch, its unit
count). A thread still inside `work` for that loop (woken late from
park, or descheduled after its last unit) passed the bound against the
larger count and claimed units the finished loop does not have. It ran
units of the new loop that were then claimed and run again, `done` was
counted past `len`, and `run` could return while a unit was still
writing buffers its caller then read and freed.

`run` now retires the old word first (its index passes no bound), then
stores the count with Release, so a thread that reads the new count sees
the retired word in its exchange, which fails; then the job, then the
new epoch.

Three tests, each failing before this change:
- a late thread in the window between the count and the epoch, with no
  pool threads (deterministic): it ran units 3..7 of the next loop, and
  every unit of that loop ran twice;
- the pool's own threads woken late from park, with the publication
  held open 200 us: in the first round, units 2..63 of a 64-unit loop
  ran twice;
- a panicking unit: `run` re-raised the panic while it held the `busy`
  guard, which poisoned it, and the next loop panicked on the
  PoisonError. The guard is now dropped before the panic is re-raised,
  and a poisoned lock is taken as it is: the last loop has finished, so
  the pool's state is whole.

All six pool tests passed in 21 runs on a loaded M3 Ultra.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw-loader-gguf: prefill and extend take the CPU passes whenever the fabric is off, and the identity tests run where the switch is off ... 0e45132e20

`prefill_on_cpu()` was `fabric().is_none() && !mpp_prefill()`. A
`fabric` build with the fabric off (the default) and
JOULECLAW_MPP_PREFILL unset therefore took the reference prefill
(attention one head at a time, experts one after another) and fed
`forward_extend` one token at a time, though nothing can leave the CPU
then: the Metal-4 path refuses without the fabric, like every other
route off the CPU. It is now `fabric().is_none()`. Logits were never
affected; the batched CPU prefill and extend were.

The identity tests compare the CPU passes with the reference passes,
which are CPU arithmetic only with the fabric off. The switch is read
once per process, and `mpp_bt_matches_neon_when_available` set
JOULECLAW_FABRIC=1 in the test process, so what those tests compared
depended on which test read the switch first; since 022310367 they
passed with nothing checked when it was on. Now no test sets the
variable in its own process. A test that needs the switch one way runs
here if it already is, otherwise alone in a child of the test binary
with the switch set, and the child must report one passed test
(`fabric_switch`). The helpers assert that the reference is CPU
arithmetic and that `forward_extend` batches, instead of returning.

Before this change, in a `rayon fabric-meter` build on macOS: with the
switch unset, the new predicate test and both extend tests failed
(prefill_on_cpu() false with the fabric off); with JOULECLAW_FABRIC=1,
all four identity tests failed the CPU-arithmetic assert. After it, all
pass both ways: with the switch on the four run in children, and with it
off the MPP test runs in a child with it on.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw-loader-gguf: a stored reference for Cohere2-MoE logits that CI checks without the checkpoint ... 97c3accb80

The five-prompt, two-Mac identity result has no stored reference, and
nothing in CI could re-check it. src/cohere2_moe_reference.txt now holds
FNV-1a hashes of every output's logits for three synthetic checkpoints:
tiny, medium, and a wide one whose decode projections are several row
chunks and whose prefill reaches the 2-D tiles. The scenario is a
prefill, decode steps, a chunk fed into the live state, and two more
steps.

The hashes come from the code before the CPU passes: commit 30c4cf1ee
(main before this pull request), its forward_prompt and forward_token
and its one-row BF16 kernel, with and without rayon (identical lines),
on macOS arm64, Linux arm64 and Linux x86_64. The platforms give
different bits, so each has its own 26 lines.

`stored_reference::logits_match_the_stored_reference` recomputes them
with the reference passes, the CPU passes and the public entry points,
then, with rayon, in child processes at 1 thread and at 3 threads with
8 KiB GEMV chunks. It passes on all three platforms. On Linux arm64 it
also passes with the glibc 2.35 libm of CI's container bound in place
of the host's 2.39; CI's linux-arm64 workspace job runs it.

BENCHMARKS.md says what this re-checks and what still needs the real
checkpoint: identity at North's own sizes and on the five prompts,
whose text and hashes are not stored, and every speed figure.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw-loader-gguf: forward_extend's doc says what it is for, not who calls it ... 3e6a60bae5

It said that a conversation adding a turn to a cached context feeds it
this way. Nothing in this repository does: its only caller is
examples/north_bench.rs, and a conversation that feeds its turn one
forward_token per token gets the same logits, more slowly. The doc now
says what the function is for and what the alternative gives.
BENCHMARKS.md now says where the follow-up-turn figures come from
(north_bench --then and --then-seq), and which of them a caller gets
by which route.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw-loader-gguf: the spin pool's run says that a unit must not start a loop on it ... b38c067054

A loop holds the pool's `busy` lock until every unit has run. A unit
that calls `run` on the same pool with more than one unit therefore
waits forever: on the caller's thread for a lock it already holds, on
a pool thread for a lock held by a caller that waits for that very
unit. No unit in this crate does so (decode's units are GEMV chunks
and attention rows); the precondition is now written where `run` is.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 3 commits 2026-11-06 07:19:26 -05:00

jouleclaw-loader-gguf: the spin pool re-raises a unit's own panic, and its epoch has 40 bits ... 2d6fe0f3df

A unit's panic reached the caller as a new panic with the pool's
message, so its payload was lost. Each panicked unit's payload is now
kept in a list rather than dropped: dropping it would run its Drop
inside the loop, where a panicking Drop would leave the unit uncounted
and the caller waiting. Once the loop has finished and the pool is
released, the caller drops the others and re-raises the first with
resume_unwind, payload and all.

The claim word was a 32-bit epoch and a 32-bit unit index. A thread
descheduled between loading the word and its exchange across exactly
2^32 - 1 loops would find the same word again in a different loop:
about 11 days of decoding at 4500 loops a second. The word is now a
40-bit epoch and a 24-bit index (about 7.7 years). A loop may have at
most 2^24 - 1 units, which `run` asserts before it takes the pool;
the decoder's largest are about 65 thousand (lm_head in 4-row chunks)
and about a million (prefill attention for a 500k-token prompt). The
safety comment states the remaining case.

The comment on `busy` now says exactly what it relies on: nothing
between taking it and releasing it panics, and why each step cannot.

Tests failing first: the unit's payload reached the caller as None (a
string from the pool); next_epoch(0xFFFF_FFFF) was 1. Both pass now,
as does the reviewer's loom model ported to the new layout (6 of 6,
the retire step still found necessary by its mutation test) and its
stress harness at 2 to 24 threads on macOS arm64, Linux arm64 and
Linux x86_64.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw-loader-gguf: the stored reference names its math library, has its generator, and runs every configuration whatever the switch ... 5e03008268

The stored logit hashes depend on the platform's math library as well
as on the code, but a mismatch blamed only the code. Each platform's
lines now carry a `math-library` line naming the library they were
made with (macOS 27.2's libsystem_m, Ubuntu GLIBC 2.39 on Linux arm64,
2.43 on Linux x86_64). A mismatch names it, says whether the ways of
running the model agree with each other (if they do, this
environment's library may round differently), and says how to tell.

examples/cohere2_reference.rs is the generator: the test's checkpoints,
scenario and hash, through forward_prompt and forward_token only, so it
runs unchanged at 30c4cf1ee. Run there, with and without rayon, it
reproduces all 78 stored lines on all three platforms, and it prints
the `math-library` line for a new one.

With JOULECLAW_FABRIC=1 in the environment, the main check ran in a
child process and the 1- and 3-thread checks never ran at all. The
test process now checks nothing itself: it starts one child per
configuration, each with the fabric off, and each child must report
its check.

Tests failing first: the mismatch message did not name the library;
a test run with JOULECLAW_FABRIC=1 reported 0 of its 3 checks. Both
pass now, with the switch unset and set.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jouleclaw-loader-gguf: BENCHMARKS.md says what each North figure compares, from the run logs ... 895778bbb3

Corrected against the per-run logs of the benchmark (final.log on both
Macs):
- The headline compared with base at its default thread count, which
  it did not say. It now gives median-against-median ratios, decode
  3.3-5.2x and prefill 5.9-10.6x, and names that baseline.
- The M3 Ultra decode median at ~600 context, 267/54 ms = 4.96x, lay
  outside the stated paired range 4.3-4.8x. The range left out a
  20.2x pair in that very column, and a second, 21.2x, at ~2700. The
  text named one at load 130-131 (it was 131, at ~2700). Both are now
  named with their loads, and the text says they are inside the
  medians.
- The M4 Max decode ranges silently left out the identity run's pairs
  (2.0x and 6.3x) while the prefill ranges kept them. They are now
  stated; the M3 prefill floor at ~2700 was 9.7, not 9.8.
- Base at 12 threads is now in the table for the M4 Max too, with
  paired ratios against it on both Macs. Base at 8 threads, its
  fastest in the earlier sweep, was never run in pairs; the text says
  so instead of implying the ratio.
- ~18 s for a turn at base's speed is an estimate (69 x 267 ms) and
  says so. A rayon pool against the spin pool was 18-24% slower on a
  quiet M4 Max (40-42 against 34 ms), not 16-24%. "Running the steps
  inside a rayon pool did not help" is about decode; prefill does run
  inside its pool, and the text says so.
- CI's glibc 2.35 giving the stored lines was shown on the host that
  made them, with 2.35's libm loaded, not by CI; the text says CI's
  run was pending.

The BF16 kernel runs 4 x 2, 2 x 4 and 1 x 4 blocks, not 4 x 4; its doc
comment and the benchmark's bullet now say so.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 1 commit 2026-11-07 03:24:45 -05:00

Merge main into jouleclaw/north-decode-speed (the flaky-test fixes, through 811ae206f) ... c762bdc388

Main gained 218 commits, among them the fixes for the four tests that
were failing on this pull request without cause in it. None of them
touches jouleclaw-loader-gguf, the jouleclaw-rs manifest or lockfile,
or the toolchain pin, and no file changed on both sides; the merge
needed no resolution.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot merged commit aa1c441f49 into main 2026-11-07 03:25:35 -05:00

dcharlot referenced this pull request from a commit 2026-11-07 03:25:41 -05:00

Merge pull request 'jouleclaw: North decode 3-6x and prefill 6-11x on the CPU, bit-identical' (#649) from jouleclaw/north-decode-speed into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!649

No description provided.