wai-quantum-writer: WAI Quantum writes a program for a task, on its own model #646

Merged

dcharlot merged 12 commits from wai/quantum-writer into main 2026-11-06 15:26:35 -05:00

Conversation 0 Commits 12 Files changed 10 +2120 -29

2026-11-06

Transaction Science

Merge main into wai/quantum-writer ...

Co-Authored-By: Claude Opus 5.5 <[email protected]>

3d8662628b

• View at this point in history

2026-11-06 15:17:48 -05:00

Transaction Science

Merge wai/quantum-check-exercise (an injected error is corrected from the qubits) into wai/quantum-writer ...

The course's tests are fifteen: #645's fourteen and this branch's one (git merged the two
raises of the count as one).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

5fbf304a41

• View at this point in history

2026-11-06 08:02:02 -05:00

Transaction Science

wai-quantum-writer: the thinking cap stays within max_new, a verified model is hashed through its mapping, and a transcript's options and maths library can be given again ...

The second review's findings in the loop and the model wrapper:
- The cap's edges. A close at the cap could take a reply past max_new (201 tokens of
  200), and one with too few tokens left gave a program cut short to check. A close now
  needs room for itself and MIN_ANSWER (64) tokens; otherwise the reply ends with no
  answer. A reply that thinks again after the close is closed again, and answer_of reads
  the text after the last close. An empty answer is no program: the model is asked
  again, the engine not asked to check nothing.
- "checked": "blake3" held only at check time: verify hashed files by name and bind
  mapped them by name again. open_verified now opens each file once, hashes it through
  the mapping the model reads (map_checked) and parses from that mapping; the config and
  tokenizer from the bytes hashed. The docs say what holds after mapping.
- Replay. --check-memory-bytes gives the command line the check's memory a service
  planned in, so every option a transcript records can be given again. The identity
  names the maths library (glibc's version, or macOS's release and build), whose
  exponentials and sines shape the logits. The matmul test now runs the dispatcher the
  projections call (matmul_bf16_auto, one row and three, and the batch).

Tests failing on fbbee6fd9 for the reasons they state:
the_cap_keeps_within_max_new_and_closes_a_reopened_thinking (201 tokens of 200),
every_recorded_option_can_be_given (the flag ignored), an_empty_answer_is_no_program
(an empty program checked). New: a_shard_is_checked_through_the_mapping_the_model_reads,
the_maths_library_is_named.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

205234abc5

• View at this point in history

2026-11-06 08:01:21 -05:00

Transaction Science

Merge wai/quantum-check-exercise (with main) into wai/quantum-writer ...

Co-Authored-By: Claude Opus 5.5 <[email protected]>

fbbee6fd98

• View at this point in history

2026-11-06 03:17:50 -05:00

Transaction Science

wai-quantum-writer: a reply's thinking is counted on its own tokens ...

thinking_tokens was the reply's tokens less its answer re-encoded, which is not the
tokens the model gave (a re-encoding splits text its own way, and the answer's markers
counted as thinking). It is now counted on the reply's tokens: up to and including the
one that ends its thinking, all of a reply whose thinking never ended, none of one that
never thought. What run 3 measures of the thinking cap rests on it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

8c3e3d09f0

• View at this point in history

2026-11-06 03:09:57 -05:00

Transaction Science

Merge wai/quantum-check-exercise (bounds said as at most n) into wai/quantum-writer ...

Co-Authored-By: Claude Opus 5.5 <[email protected]>

225647f48b

• View at this point in history

2026-11-06 03:09:57 -05:00

Transaction Science

wai-quantum-writer: one task digest names one check, the model loads exactly its manifest, and a reply's thinking is capped ...

The loop, as reviewed:
- A task is read in its canonical form (RFC 8785, what task_digest is taken over) and
  held to engine::course::exercise_form before the model is asked anything. The same
  task with its keys in another order, or a bound written 0.0, was two prompts and two
  checks under one digest; a job asking for a repair, a vacuous bound or an unknown
  constraint went to the model.
- The transcript records check_memory_bytes, the memory each check's run is planned in,
  which decides what a check runs: the command line and the service used different caps
  and their transcripts did not say so.
- The replay claim is scoped: the model's identity names the arithmetic it ran in
  (arch, os, and the BF16 matmul kernel: NEON fused multiply-add on aarch64, a scalar
  loop elsewhere), and a test holds the named kernel to the one that runs. The same
  files, task, options and engine give the same bytes in that arithmetic.

The model wrapper:
- CodeModel::open loads exactly the files wai-model.json lists, in its order, and refuses
  a manifest that lists none and a directory holding a model file it does not list or
  lacking one it lists (a stray shard sorting first was bound in place of the listed
  one). It checks sizes, and says so in the identity ("checked": "sizes");
  open_verified (and --verify) reads every byte first and says "blake3".
- answer_of drops <|START_TEXT|> and cuts at <|END_TEXT|>, which landed in the program.

And the measured "no answer" (run 2: 23 of 43 attempts spent the whole budget thinking):
a reply still thinking at its cap (Options::thinking_cap, two thirds of max_new unless
--think-max says) has <|END_THINKING|><|START_TEXT|> fed for it and decodes its answer in
the tokens left. The transcript records the cap and, per attempt, thinking_closed;
`course` prints how many replies gave no program and how many were closed. The decode
loop is a function over a small trait, so a test drives it with a model that thinks
for ever.

Each loop test failed on 63fb4a39c for the reason it states (a malformed task was
written for; one task, two prompts), and the coder tests for theirs (the markers were
in the answer; an unlisted shard was taken). The course's case names given as one
string are shown by it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

64022d1b18

• View at this point in history

2026-11-06 03:00:47 -05:00

Transaction Science

Merge wai/quantum-check-exercise (check_exercise reads its exercise first) into wai/quantum-writer ...

The course tests are fourteen: wai-quantum-docs.yml names each of both branches'.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

49d30ddf42

• View at this point in history

2026-11-06 02:48:47 -05:00

2026-11-05

Transaction Science

Merge wai/quantum-check-exercise (with main) into wai/quantum-writer ...

The branch below this one now carries main's two newer commits (the marks,
#653). They touch nothing this branch does. No conflict.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

63fb4a39c8

• View at this point in history

2026-11-05 17:29:40 -05:00

Transaction Science

wai-quantum-docs.yml: the course's tests are eleven, and this branch's is required by name ...

The workflow pins how many course tests pass and names each one, since a
module filter is satisfied by any one test. This branch adds
any_exercise_is_shown_as_a_lesson_shows_its_own to engine::course, and
the pin it now inherits from the branch below counts ten. So the step
failed here with all eleven passing ("test result: ok. 11 passed" in the
job's log, then "the course's tests did not all pass").

The count is eleven and the new name is the last in the list, the same
lines the branch stacked above this one already holds, so the two merge
as one change.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

0d5f9b871a

• View at this point in history

2026-11-05 17:04:49 -05:00

Transaction Science

Merge wai/quantum-check-exercise (with main) into wai/quantum-writer ...

The branch below this one now carries main's fourteen newer commits, its
count-of-ten pin for the course's tests, and its changelog entry as a
file of wai/changes/.

One conflict, wai/CHANGELOG.md: this branch had added two entries to the
head of Unreleased, the one it inherited and its own. The file is main's
again. The inherited entry is already the file the branch below added,
byte for byte, and this branch's own entry moves, word for word, to
wai/changes/2026-11-05-quantum-writer-writes-program-for-task.md.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

b566a06e7e

• View at this point in history

2026-11-05 17:04:27 -05:00

Transaction Science

wai-quantum-writer: WAI Quantum writes a program for a task, on its own model ...

The loop the job service will write programs with, as a crate of its own. The task is an exercise
in the course's form (wai.quantum.exercise/1). A model writes a program, and the engine checks it
against the task exactly as it checks a lesson's answer (engine::course::check_exercise). What went
wrong goes back to the model in the engine's own words, until a program passes or the rounds run
out.

- The code model (src/coder.rs): North Mini Code 1.0 (Apache-2.0) on the pure-Rust runtime
  (jouleclaw-loader-gguf).
  - Decoding is greedy, so the same model files, task and engine give the same programs.
  - The conversation stays in the model's cache: a repair round feeds only the new turn, where
    the measurement harness re-read the whole conversation each round.
  - The loop takes the answer after the model's thinking, so thinking text is never checked as a
    program, and a reply that never leaves its thinking is asked again.
- The model's identity: the digest of a manifest of its files (wai.quantum.model/1,
  wai-model.json). open() checks each file's size against it, and verify reads every byte.
- The transcript (wai.quantum.write/1): the task, model and prompt (version, system and first-turn
  digests), the decoding, the engine's version, and each attempt (program, digest, the engine's
  check, tokens). Timing is returned beside it, since a replay does not reproduce time.
- The engine:
  - course::task_of shows any exercise as a lesson shows its own, its title and task given as one
    string or one per locale;
  - course::exercise gives a lesson's exercise document;
  - lesson() now builds its fields through the same helpers, and a test holds every lesson's
    fields in both locales equal to task_of's.
- The command line: wai-quantum-writer write | course | manifest | verify.
- CI: wai-quantum-cli.yml tests the writer with a scripted model, with and without the model
  compiled in, plus clippy. The workflow now also runs for changes to the writer and to the
  loader crate it depends on.

Tests (all run on the build machines):
- wai-quantum-writer: 7 pass, default features and without them; clippy -D warnings is clean.
- engine::course: 11 pass.
- scripts/check-lockfiles.sh: 61 lockfiles, all current.

Stacked on #645 (check_exercise).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

ef4d02e547

• View at this point in history

2026-11-05 08:03:56 -05:00