wai-quantum-writer: WAI Quantum writes a program for a task, on its own model #646

Merged

dcharlot merged 12 commits from wai/quantum-writer into main 2026-11-06 15:26:35 -05:00

Conversation 0 Commits 12 Files changed 10 +2120 -29

dcharlot commented 2026-11-05 08:04:39 -05:00

Owner

Copy link

The loop the job service will use to write a program for a task, on WAI's own model and runtime. Stacked on #645 (check_exercise): this branch carries #645's commit, and its watcher starts once #645 merges.

What it does

A task is an exercise in the course's form (wai.quantum.exercise/1): what the program must do (an intent, or one for each case), its constraints, its job, and a starter program. The loop runs in rounds:

1. The model writes a program.
2. The engine checks it exactly as it checks a lesson's answer.
3. If the check fails, what went wrong goes back to the model in the engine's own words, with where to look, for another attempt.

It stops at the first program that passes, or when the rounds run out (four repairs by default).

• The code model (src/coder.rs): North Mini Code 1.0 (Apache-2.0) on the pure-Rust runtime (jouleclaw-loader-gguf).

  • Decoding is greedy, so the same model files, task and engine give the same programs in the same arithmetic. The model's identity names that arithmetic (architecture, operating system, matmul kernel); on another, a near tie can go the other way.
  • The conversation stays in the model's cache, so a repair round feeds only the new turn instead of reading the whole conversation again.
  • The program is taken from the answer that follows the model's thinking. If a reply never leaves its thinking, the model is asked again. Both came out of measuring the loop on the course; see below.

• The model's identity: a manifest of its files (wai.quantum.model/1, wai-model.json), with each file's name, size and BLAKE3 digest. The model is named by that manifest's digest. open loads exactly the files the manifest lists and checks their sizes; open_verified (and --verify) reads every byte first. The identity says which ("checked").

• The transcript (wai.quantum.write/1) records:

  • the task and its digest, the model's identity, and the prompt (its version, and the digests of the system text and the first turn);
  • the decoding, and the engine's version;
  • every attempt: the program, its digest, the engine's full check, its tokens and its thinking tokens.

  Timing is returned beside the transcript rather than inside it, since a replay does not reproduce time.

• The command line: wai-quantum-writer write --model DIR TASK.json, course --model DIR --out DIR (the measurement below), manifest DIR, verify DIR.

• The engine:

  • course::task_of shows any exercise the way a lesson shows its own. Its title and task can be one string or one per locale.
  • course::exercise(id) returns a lesson's exercise document.
  • lesson() now builds its fields through the same helpers.

Why on our own model

WAI Quantum is its own stack. A program that WAI writes for a task should not depend on another company's model or agent. The runtime is ours and its decode is greedy, so a written program can be replayed byte for byte from the model's digest, on a host with the same arithmetic.

What measuring it on the course showed

I ran an earlier version of this loop as a harness over the course's 21 exercises, on the build host's CPU.

• First run: 5/7/9/10/11 of 21 passed after 0/1/2/3/4 repairs. That run understated the model.
  • Its 700-token budget often ran out while the model was still thinking.
  • 16 of 73 attempts handed the thinking text to the engine as if it were the program.
  • Every round re-read the whole conversation from the start.
• Second run: it takes the answer only, budgets 3000 tokens and keeps the cache. When this PR was opened, the first eight exercises had all passed at once, including three that had needed two to four repairs before. It is still running, and its results will go into this PR when it ends.
• Speed is the open problem: decode runs at about 4–5 tokens per second on the CPU. The GPU path gives identical tokens but is slower for this model. Work on the CPU decode is under way separately.

Review fixes (2026-11-06)

A review of the write loop found these in the loop and the model wrapper. Fixed here:

• One task digest names one prompt and one check. The loop reads a task in its canonical form (RFC 8785, what task_digest is taken over), and holds it to engine::course::exercise_form (#645) before the model is asked anything. Before, the same task with its keys in another order, or a bound written 0.0, was two prompts and two checks under one digest. A task whose job asked for a repair, or whose bound held vacuously, went to the model.
• The transcript records what decides a check: check_memory_bytes, the memory each check's run is planned in. The command line and the service planned in different memory, and the transcripts did not say so.
• The replay claim is scoped. The model's identity names the arithmetic it ran in: arch, os and matmul, which is NEON fused multiply-add on aarch64 and a scalar loop elsewhere. A test holds the named kernel to the one that runs. The claim in the crate's docs, its description and the changelog is now "the same bytes in the same arithmetic".
• The model loads exactly its manifest. open binds the files the manifest lists, in its order. It refuses a manifest that lists none, and a directory that holds a model file the manifest does not list or lacks one it lists. Before, a stray shard that sorted first was bound in place of the listed one, under the old digest. The identity says how far the files were checked: "sizes", or "blake3" with open_verified / --verify.
• The answer's markers stay out of the program: answer_of drops <|START_TEXT|> and cuts at <|END_TEXT|>.
• A reply's thinking is counted on its own tokens, up to the one that ends it. Before, the count was the reply less its answer re-encoded.

Second review (2026-11-06)

• The cap's edges.
  • A close at the cap could take a reply past max_new (201 tokens of 200).
  • A close with too few tokens left gave a cut-off program to check. A close now needs room for itself and 64 tokens (MIN_ANSWER); otherwise the reply ends with no answer.
  • A reply that thinks again after the close is closed again, and the answer is what follows its last close.
  • An empty answer is no program.
  • Tests the_cap_keeps_within_max_new_and_closes_a_reopened_thinking and an_empty_answer_is_no_program both failed on fbbee6fd9 for the reasons they state.
• "checked": "blake3" held only at check time. open_verified now opens each file once, hashes it through the mapping the model reads (map_checked), and parses from that same mapping. It reads the config and the tokenizer from the bytes it hashed. The docs say what holds after mapping: a file renamed over is not seen; a file written in place can change what the mapping reads. Test a_shard_is_checked_through_the_mapping_the_model_reads.
• Replay.
  • --check-memory-bytes gives the command line the check's memory a service planned in. Test every_recorded_option_can_be_given, which failed on fbbee6fd9 because the flag was ignored.
  • The identity names the maths library (libm: glibc's version, or macOS's release and build). Test the_maths_library_is_named.
  • The matmul test now runs the dispatcher the projections call: matmul_bf16_auto for one row and three, and the batch.

The "no answer" replies: a thinking cap

Run 2 (21 lessons, 3,000 tokens, greedy) passed 11 at once, 15 after one repair and 18 after two, then stayed flat. The three it never passed (phase-estimation, bit-flip-code, toffoli) all ended in no answer. 23 of its 43 attempts spent the whole budget thinking.

A reply still thinking at its cap now has <|END_THINKING|><|START_TEXT|> fed for it, and decodes its answer in the tokens left. The cap is Options::thinking_cap: two thirds of max_new (2,000 of 3,000), or --think-max N. The transcript records the cap (decoding.think_max) and, for each attempt, thinking_closed. course prints how many replies gave no program and how many had their thinking closed. The decode loop is a function over a small trait, so a test drives it with a model that thinks for ever. It is not yet measured on the real model; how to measure it:

1. On the host with the checkpoint, build the writer and run wai-quantum-writer course --model DIR --out run3-off --think-max 3000. A cap equal to max_new never closes, so this is run 2's setting on this code. Re-run it rather than compare with run 2: canonical tasks can order a lesson's prompt differently.
2. Run wai-quantum-writer course --model DIR --out run3-cap, at the default cap of 2,000. Optionally run 1,500 and 2,500 as well.
3. Compare the printed "passed after r repair(s)" lines and the "replies: N, of which U gave no program and C had their thinking closed" line. From each results.jsonl, compare per lesson: answered, thinking_closed, thinking_tokens and passed for each attempt, and timing.
4. Watch two things. The three lessons above should now answer. A lesson that passed at once without the cap and fails with it would mean the cap cut thinking it needed.

Tests

• tests/write.rs runs the loop with a scripted model, so no checkpoint is needed. It covers:
  • a program that passes at once;
  • a repair driven by the engine's words;
  • a reply that never leaves its thinking;
  • rounds that run out without a pass;
  • identical transcript bytes for the same inputs;
  • a task with nothing to check;
  • extracting the program from an answer;
  • new: one task digest naming one prompt and one check, and a malformed task refused before the model is asked. Each failed on 63fb4a39c for the reason it states: one task, two prompts; a task with {"t_count": 0} was written for.
  • new: the transcript recording the check's memory and the thinking cap.
• src/coder.rs (new, with the model compiled in, no checkpoint):
  • the thinking cap closing a reply that would think for ever;
  • the answer's markers;
  • a model directory holding exactly its manifest's files. Both of the last two failed on 63fb4a39c: the markers were in the answer, and an unlisted shard was taken.
  • the named matmul kernel being the one that runs;
  • the thinking count.
• Counts, Linux x86_64 and macOS aarch64: 20 with the model compiled in (8 in coder, 1 in the command line, 11 in tests/write.rs), 11 without. Clippy with -D warnings is clean in both configurations.
• engine::course: 15 pass, including any_exercise_is_shown_as_a_lesson_shows_its_own (every lesson, both locales, and a case named with one string).
• The pin: wai-quantum-docs.yml counts the course's fifteen tests and requires each by name.
• Changelog: a file of its own, wai/changes/2026-11-05-quantum-writer-writes-program-for-task.md, beside #645's. wai/CHANGELOG.md is main's.
• CI: wai-quantum-cli.yml now runs the writer's tests and clippy. It also runs when the writer or jouleclaw-loader-gguf changes.

Not covered here: a run of CodeModel against the real checkpoint (57 GB), with or without the cap; the steps above measure it.

🤖 Generated with Claude Code

The loop the job service will use to write a program for a task, on WAI's own model and runtime. **Stacked on #645** (`check_exercise`): this branch carries #645's commit, and its watcher starts once #645 merges. ## What it does A task is an exercise in the course's form (`wai.quantum.exercise/1`): what the program must do (an intent, or one for each case), its constraints, its job, and a starter program. The loop runs in rounds: 1. The model writes a program. 2. The engine checks it exactly as it checks a lesson's answer. 3. If the check fails, what went wrong goes back to the model in the engine's own words, with where to look, for another attempt. It stops at the first program that passes, or when the rounds run out (four repairs by default). - **The code model** (`src/coder.rs`): North Mini Code 1.0 (Apache-2.0) on the pure-Rust runtime (`jouleclaw-loader-gguf`). - Decoding is greedy, so the same model files, task and engine give the same programs in the same arithmetic. The model's identity names that arithmetic (architecture, operating system, matmul kernel); on another, a near tie can go the other way. - The conversation stays in the model's cache, so a repair round feeds only the new turn instead of reading the whole conversation again. - The program is taken from the answer that follows the model's thinking. If a reply never leaves its thinking, the model is asked again. Both came out of measuring the loop on the course; see below. - **The model's identity**: a manifest of its files (`wai.quantum.model/1`, `wai-model.json`), with each file's name, size and BLAKE3 digest. The model is named by that manifest's digest. `open` loads exactly the files the manifest lists and checks their sizes; `open_verified` (and `--verify`) reads every byte first. The identity says which (`"checked"`). - **The transcript** (`wai.quantum.write/1`) records: - the task and its digest, the model's identity, and the prompt (its version, and the digests of the system text and the first turn); - the decoding, and the engine's version; - every attempt: the program, its digest, the engine's full check, its tokens and its thinking tokens. Timing is returned beside the transcript rather than inside it, since a replay does not reproduce time. - **The command line**: `wai-quantum-writer write --model DIR TASK.json`, `course --model DIR --out DIR` (the measurement below), `manifest DIR`, `verify DIR`. - **The engine**: - `course::task_of` shows any exercise the way a lesson shows its own. Its title and task can be one string or one per locale. - `course::exercise(id)` returns a lesson's exercise document. - `lesson()` now builds its fields through the same helpers. ## Why on our own model WAI Quantum is its own stack. A program that WAI writes for a task should not depend on another company's model or agent. The runtime is ours and its decode is greedy, so a written program can be replayed byte for byte from the model's digest, on a host with the same arithmetic. ## What measuring it on the course showed I ran an earlier version of this loop as a harness over the course's 21 exercises, on the build host's CPU. - **First run: 5/7/9/10/11 of 21 passed after 0/1/2/3/4 repairs.** That run understated the model. - Its 700-token budget often ran out while the model was still thinking. - 16 of 73 attempts handed the thinking text to the engine as if it were the program. - Every round re-read the whole conversation from the start. - **Second run:** it takes the answer only, budgets 3000 tokens and keeps the cache. When this PR was opened, the first eight exercises had all passed at once, including three that had needed two to four repairs before. It is still running, and its results will go into this PR when it ends. - **Speed is the open problem:** decode runs at about 4–5 tokens per second on the CPU. The GPU path gives identical tokens but is slower for this model. Work on the CPU decode is under way separately. ## Review fixes (2026-11-06) A review of the write loop found these in the loop and the model wrapper. Fixed here: - **One task digest names one prompt and one check.** The loop reads a task in its canonical form (RFC 8785, what `task_digest` is taken over), and holds it to `engine::course::exercise_form` (#645) before the model is asked anything. Before, the same task with its keys in another order, or a bound written `0.0`, was two prompts and two checks under one digest. A task whose job asked for a repair, or whose bound held vacuously, went to the model. - **The transcript records what decides a check:** `check_memory_bytes`, the memory each check's run is planned in. The command line and the service planned in different memory, and the transcripts did not say so. - **The replay claim is scoped.** The model's identity names the arithmetic it ran in: `arch`, `os` and `matmul`, which is NEON fused multiply-add on aarch64 and a scalar loop elsewhere. A test holds the named kernel to the one that runs. The claim in the crate's docs, its description and the changelog is now "the same bytes in the same arithmetic". - **The model loads exactly its manifest.** `open` binds the files the manifest lists, in its order. It refuses a manifest that lists none, and a directory that holds a model file the manifest does not list or lacks one it lists. Before, a stray shard that sorted first was bound in place of the listed one, under the old digest. The identity says how far the files were checked: `"sizes"`, or `"blake3"` with `open_verified` / `--verify`. - **The answer's markers stay out of the program:** `answer_of` drops `<|START_TEXT|>` and cuts at `<|END_TEXT|>`. - **A reply's thinking is counted on its own tokens**, up to the one that ends it. Before, the count was the reply less its answer re-encoded. ## Second review (2026-11-06) - **The cap's edges.** - A close at the cap could take a reply past `max_new` (201 tokens of 200). - A close with too few tokens left gave a cut-off program to check. A close now needs room for itself and 64 tokens (`MIN_ANSWER`); otherwise the reply ends with no answer. - A reply that thinks again after the close is closed again, and the answer is what follows its last close. - An empty answer is no program. - Tests `the_cap_keeps_within_max_new_and_closes_a_reopened_thinking` and `an_empty_answer_is_no_program` both failed on fbbee6fd9 for the reasons they state. - **`"checked": "blake3"` held only at check time.** `open_verified` now opens each file once, hashes it through the mapping the model reads (`map_checked`), and parses from that same mapping. It reads the config and the tokenizer from the bytes it hashed. The docs say what holds after mapping: a file renamed over is not seen; a file written in place can change what the mapping reads. Test `a_shard_is_checked_through_the_mapping_the_model_reads`. - **Replay.** - `--check-memory-bytes` gives the command line the check's memory a service planned in. Test `every_recorded_option_can_be_given`, which failed on fbbee6fd9 because the flag was ignored. - The identity names the maths library (`libm`: glibc's version, or macOS's release and build). Test `the_maths_library_is_named`. - The matmul test now runs the dispatcher the projections call: `matmul_bf16_auto` for one row and three, and the batch. ## The "no answer" replies: a thinking cap Run 2 (21 lessons, 3,000 tokens, greedy) passed 11 at once, 15 after one repair and 18 after two, then stayed flat. The three it never passed (`phase-estimation`, `bit-flip-code`, `toffoli`) all ended in no answer. 23 of its 43 attempts spent the whole budget thinking. A reply still thinking at its cap now has `<|END_THINKING|><|START_TEXT|>` fed for it, and decodes its answer in the tokens left. The cap is `Options::thinking_cap`: two thirds of `max_new` (2,000 of 3,000), or `--think-max N`. The transcript records the cap (`decoding.think_max`) and, for each attempt, `thinking_closed`. `course` prints how many replies gave no program and how many had their thinking closed. The decode loop is a function over a small trait, so a test drives it with a model that thinks for ever. It is not yet measured on the real model; how to measure it: 1. On the host with the checkpoint, build the writer and run `wai-quantum-writer course --model DIR --out run3-off --think-max 3000`. A cap equal to `max_new` never closes, so this is run 2's setting on this code. Re-run it rather than compare with run 2: canonical tasks can order a lesson's prompt differently. 2. Run `wai-quantum-writer course --model DIR --out run3-cap`, at the default cap of 2,000. Optionally run 1,500 and 2,500 as well. 3. Compare the printed "passed after r repair(s)" lines and the "replies: N, of which U gave no program and C had their thinking closed" line. From each `results.jsonl`, compare per lesson: `answered`, `thinking_closed`, `thinking_tokens` and `passed` for each attempt, and `timing`. 4. Watch two things. The three lessons above should now answer. A lesson that passed at once without the cap and fails with it would mean the cap cut thinking it needed. ## Tests - **`tests/write.rs`** runs the loop with a scripted model, so no checkpoint is needed. It covers: - a program that passes at once; - a repair driven by the engine's words; - a reply that never leaves its thinking; - rounds that run out without a pass; - identical transcript bytes for the same inputs; - a task with nothing to check; - extracting the program from an answer; - **new:** one task digest naming one prompt and one check, and a malformed task refused before the model is asked. Each failed on 63fb4a39c for the reason it states: one task, two prompts; a task with `{"t_count": 0}` was written for. - **new:** the transcript recording the check's memory and the thinking cap. - **`src/coder.rs`** (new, with the model compiled in, no checkpoint): - the thinking cap closing a reply that would think for ever; - the answer's markers; - a model directory holding exactly its manifest's files. Both of the last two failed on 63fb4a39c: the markers were in the answer, and an unlisted shard was taken. - the named matmul kernel being the one that runs; - the thinking count. - **Counts, Linux x86_64 and macOS aarch64:** 20 with the model compiled in (8 in `coder`, 1 in the command line, 11 in `tests/write.rs`), 11 without. Clippy with `-D warnings` is clean in both configurations. - **`engine::course`**: 15 pass, including `any_exercise_is_shown_as_a_lesson_shows_its_own` (every lesson, both locales, and a case named with one string). - **The pin**: `wai-quantum-docs.yml` counts the course's fifteen tests and requires each by name. - **Changelog**: a file of its own, `wai/changes/2026-11-05-quantum-writer-writes-program-for-task.md`, beside #645's. `wai/CHANGELOG.md` is main's. - **CI**: `wai-quantum-cli.yml` now runs the writer's tests and clippy. It also runs when the writer or `jouleclaw-loader-gguf` changes. Not covered here: a run of `CodeModel` against the real checkpoint (57 GB), with or without the cap; the steps above measure it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 2 commits 2026-11-05 08:04:39 -05:00

wai-quantum: any exercise is checked as a lesson is (engine::course::check_exercise) ... 898793622e

The course's check held a program to a lesson's exercise: its cases and intents, its constraints,
the engine's verdict at each, and what went wrong in words. Only the lesson lookup tied it to the
course. check_exercise takes the exercise as a document in the course's own form
(wai.quantum.exercise/1) and checks a program against it the same way, with `lesson` null.

It is the first step toward the job service writing a program for an intent: each attempt is held
to the task as a lesson's answer is, and what went wrong is the feedback for the next attempt.

check now finds its lesson and calls the same function, so a lesson's check is the same bytes as
before. A new test holds both:
- a lesson's own exercise, given as a document, checks byte for byte as the lesson does, for a
  right and a wrong program;
- an exercise written from nothing but a distribution passes a Bell pair by proof and fails a
  program without its CNOT, naming the distribution.

The course's ten tests pass (cargo test --features full --lib engine::course).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai-quantum-writer: WAI Quantum writes a program for a task, on its own model ... ef4d02e547

The loop the job service will write programs with, as a crate of its own. The task is an exercise
in the course's form (wai.quantum.exercise/1). A model writes a program, and the engine checks it
against the task exactly as it checks a lesson's answer (engine::course::check_exercise). What went
wrong goes back to the model in the engine's own words, until a program passes or the rounds run
out.

- The code model (src/coder.rs): North Mini Code 1.0 (Apache-2.0) on the pure-Rust runtime
  (jouleclaw-loader-gguf).
  - Decoding is greedy, so the same model files, task and engine give the same programs.
  - The conversation stays in the model's cache: a repair round feeds only the new turn, where
    the measurement harness re-read the whole conversation each round.
  - The loop takes the answer after the model's thinking, so thinking text is never checked as a
    program, and a reply that never leaves its thinking is asked again.
- The model's identity: the digest of a manifest of its files (wai.quantum.model/1,
  wai-model.json). open() checks each file's size against it, and verify reads every byte.
- The transcript (wai.quantum.write/1): the task, model and prompt (version, system and first-turn
  digests), the decoding, the engine's version, and each attempt (program, digest, the engine's
  check, tokens). Timing is returned beside it, since a replay does not reproduce time.
- The engine:
  - course::task_of shows any exercise as a lesson shows its own, its title and task given as one
    string or one per locale;
  - course::exercise gives a lesson's exercise document;
  - lesson() now builds its fields through the same helpers, and a test holds every lesson's
    fields in both locales equal to task_of's.
- The command line: wai-quantum-writer write | course | manifest | verify.
- CI: wai-quantum-cli.yml tests the writer with a scripted model, with and without the model
  compiled in, plus clippy. The workflow now also runs for changes to the writer and to the
  loader crate it depends on.

Tests (all run on the build machines):
- wai-quantum-writer: 7 pass, default features and without them; clippy -D warnings is clean.
- engine::course: 11 pass.
- scripts/check-lockfiles.sh: 61 lockfiles, all current.

Stacked on #645 (check_exercise).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-11-05 10:08:14 -05:00

wai-quantum-service: POST /v1/write and write_circuit, a program written for a task on the service's own model (stacked on #646) #648

dcharlot referenced this pull request from a commit 2026-11-05 11:09:19 -05:00

wai-quantum-docs.yml: the stack's course tests are required by name, and their count is eleven

dcharlot added 7 commits 2026-11-05 18:05:14 -05:00

wai-quantum-docs.yml: the course's new test is required by name, and its count is ten ... af83fbbe9f

The workflow pins how many course tests pass and names each one, since a module filter is satisfied
by any one test. check_exercise added a tenth, so the step failed with every test passing.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge main into wai/quantum-check-exercise ... 4663a66178

Main gained fourteen commits since this branch was cut, among them the
changelog's new layout (one entry a file under wai/changes/). Nothing
conflicted; the entry this branch added to the head of wai/CHANGELOG.md
moves to a file in the next commit.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

wai-quantum: check_exercise's changelog entry is a file of wai/changes/ ... eef9fa1095

Main now takes one file an entry (wai/changes/README.md), and its
repository checks fail a pull request opened since that adds to the head
of wai/CHANGELOG.md, which is what this branch did. The entry moves, word
for word, to wai/changes/2026-11-05-quantum-any-exercise-checked-as-lesson.md,
and wai/CHANGELOG.md is main's again.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

Merge wai/quantum-check-exercise (with main) into wai/quantum-writer ... b566a06e7e

The branch below this one now carries main's fourteen newer commits, its
count-of-ten pin for the course's tests, and its changelog entry as a
file of wai/changes/.

One conflict, wai/CHANGELOG.md: this branch had added two entries to the
head of Unreleased, the one it inherited and its own. The file is main's
again. The inherited entry is already the file the branch below added,
byte for byte, and this branch's own entry moves, word for word, to
wai/changes/2026-11-05-quantum-writer-writes-program-for-task.md.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

wai-quantum-docs.yml: the course's tests are eleven, and this branch's is required by name ... 0d5f9b871a

The workflow pins how many course tests pass and names each one, since a
module filter is satisfied by any one test. This branch adds
any_exercise_is_shown_as_a_lesson_shows_its_own to engine::course, and
the pin it now inherits from the branch below counts ten. So the step
failed here with all eleven passing ("test result: ok. 11 passed" in the
job's log, then "the course's tests did not all pass").

The count is eleven and the new name is the last in the list, the same
lines the branch stacked above this one already holds, so the two merge
as one change.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

Merge main into wai/quantum-check-exercise ... 13015f77af

Main's two newer commits (the marks, #653) touch nothing this branch does.
No conflict.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

Merge wai/quantum-check-exercise (with main) into wai/quantum-writer ... 63fb4a39c8

The branch below this one now carries main's two newer commits (the marks,
#653). They touch nothing this branch does. No conflict.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

dcharlot referenced this pull request 2026-11-06 03:23:54 -05:00

ci: the wai-quantum legs build the commit they test, and fail when they did not #666

dcharlot added 8 commits 2026-11-06 03:25:35 -05:00

wai-quantum: check_exercise reads its exercise first, binds every case's inputs, and counts T gates as gates ... 4dcd8ff4a5

A check of an exercise that is no lesson's held vacuously where it read nothing, and
the writer makes that check the only gate on a pass. Now:

- engine::course::exercise_form holds a document to a lesson's form as far as a check
  reads it, and check_exercise refuses what it refuses before anything runs: a bound
  that is not whole numbers, a constraint or field the course does not name, a job
  beyond shots, seed and noise (a non-object job panicked the check; a repair or
  dyadic_bits made the engine prove another program than the one recorded), more than
  MOST_CASES (16) cases or MOST_INTENTS (8) intents. read_exercise shares the same
  reading, and a lesson's cases are still named in every locale.
- A case's inputs bind: a program that does not declare one is refused in that case
  (Q114), where it ran without it (the bit-flip lesson passed with no flip). A case
  whose intent is null takes the exercise's, and a case named with one string is
  shown by it.
- The constraints hold in every case's program, not the first one's, and a broken one
  names its case.
- t_count counts T gates, not names: p and rz by an odd multiple of pi/4 count one, a
  Clifford gate none, and any other gate does not hold to a T count. A three-qubit gate
  does not hold to a two-qubit count. The t-count lesson passed with every T written as
  p(pi/4).

Three tests, each failing on the branch's previous head for the reason it states
(13015f77a: a {"t_count": 0} bound was checked and passed; the no-flip program passed
the bit-flip lesson; the respelled t-count answer passed): course tests 10 -> 13 in
wai-quantum-docs.yml, run on Linux x86_64 and macOS aarch64.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge wai/quantum-check-exercise (check_exercise reads its exercise first) into wai/quantum-writer ... 49d30ddf42

The course tests are fourteen: wai-quantum-docs.yml names each of both branches'.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai-quantum-writer: one task digest names one check, the model loads exactly its manifest, and a reply's thinking is capped ... 64022d1b18

The loop, as reviewed:
- A task is read in its canonical form (RFC 8785, what task_digest is taken over) and
  held to engine::course::exercise_form before the model is asked anything. The same
  task with its keys in another order, or a bound written 0.0, was two prompts and two
  checks under one digest; a job asking for a repair, a vacuous bound or an unknown
  constraint went to the model.
- The transcript records check_memory_bytes, the memory each check's run is planned in,
  which decides what a check runs: the command line and the service used different caps
  and their transcripts did not say so.
- The replay claim is scoped: the model's identity names the arithmetic it ran in
  (arch, os, and the BF16 matmul kernel: NEON fused multiply-add on aarch64, a scalar
  loop elsewhere), and a test holds the named kernel to the one that runs. The same
  files, task, options and engine give the same bytes in that arithmetic.

The model wrapper:
- CodeModel::open loads exactly the files wai-model.json lists, in its order, and refuses
  a manifest that lists none and a directory holding a model file it does not list or
  lacking one it lists (a stray shard sorting first was bound in place of the listed
  one). It checks sizes, and says so in the identity ("checked": "sizes");
  open_verified (and --verify) reads every byte first and says "blake3".
- answer_of drops <|START_TEXT|> and cuts at <|END_TEXT|>, which landed in the program.

And the measured "no answer" (run 2: 23 of 43 attempts spent the whole budget thinking):
a reply still thinking at its cap (Options::thinking_cap, two thirds of max_new unless
--think-max says) has <|END_THINKING|><|START_TEXT|> fed for it and decodes its answer in
the tokens left. The transcript records the cap and, per attempt, thinking_closed;
`course` prints how many replies gave no program and how many were closed. The decode
loop is a function over a small trait, so a test drives it with a model that thinks
for ever.

Each loop test failed on 63fb4a39c for the reason it states (a malformed task was
written for; one task, two prompts), and the coder tests for theirs (the markers were
in the answer; an unlisted shard was taken). The course's case names given as one
string are shown by it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai-quantum: a T count, a two-qubit count and an estimate's bound are "at most n" alone ... 423af2a325

The course words these bounds as "at most n" (constraint.t_count, two_qubit, estimate),
and the check read a min too: {"t_count": {"min": 2}} showed the model "at most ? T and
T† gates" while it failed every program with fewer than two. The reader now takes
{"max": n} alone for them (engine::course::at_most), and exercise.v1.json says so
($defs/at_most), with the T count's description saying how T gates are counted. No
lesson uses a min there.

Found sweeping the review's family of claims the check makes that the code does not keep.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge wai/quantum-check-exercise (bounds said as at most n) into wai/quantum-writer ... 225647f48b

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai-quantum-writer: a reply's thinking is counted on its own tokens ... 8c3e3d09f0

thinking_tokens was the reply's tokens less its answer re-encoded, which is not the
tokens the model gave (a re-encoding splits text its own way, and the answer's markers
counted as thinking). It is now counted on the reply's tokens: up to and including the
one that ends its thinking, all of a reply whose thinking never ended, none of one that
never thought. What run 3 measures of the thinking cap rests on it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge main into wai/quantum-check-exercise ... ba59c776eb

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge wai/quantum-check-exercise (with main) into wai/quantum-writer ... fbbee6fd98

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 3 commits 2026-11-06 08:19:30 -05:00

wai-quantum: an injected error is corrected from the qubits, never read: the error-correction lessons check behaviour ... 12496404b5

The second review found finding 1 closed only for undeclared inputs. A program that
declared `input uint err;` and never used it passed the bit-flip lesson at proof, and
one that branched on the classical err passed the syndrome lesson.

A new constraint, `inject`, names the input whose value is the qubit an error flips.
The cases' programs may then differ in that one gate alone, an x on the qubit the case
names, every other gate, measurement and step the same (course::injected). So the error
is in the quantum state, the program cannot read it classically, and what finds and
corrects it is the program's quantum operations, the same in every case. A program
that never applies the error, reads it elsewhere, or flips another qubit fails, saying
which. A second, `qubits` ({"max": n}), closes the last way round: an answer kept on a
qubit no error reaches. bit-flip-code and syndrome set both; their solutions and wrong
answers keep their results. The exercise schema names both, and a case set an
injection cannot be made from is refused.

Test an_injected_error_is_corrected_from_the_qubits_and_never_read, failing on
ba59c776e for the reason it states (the declared, unused input passed bit-flip-code).
Course tests 13 -> 14 in wai-quantum-docs.yml.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai-quantum-writer: the thinking cap stays within max_new, a verified model is hashed through its mapping, and a transcript's options and maths library can be given again ... 205234abc5

The second review's findings in the loop and the model wrapper:
- The cap's edges. A close at the cap could take a reply past max_new (201 tokens of
  200), and one with too few tokens left gave a program cut short to check. A close now
  needs room for itself and MIN_ANSWER (64) tokens; otherwise the reply ends with no
  answer. A reply that thinks again after the close is closed again, and answer_of reads
  the text after the last close. An empty answer is no program: the model is asked
  again, the engine not asked to check nothing.
- "checked": "blake3" held only at check time: verify hashed files by name and bind
  mapped them by name again. open_verified now opens each file once, hashes it through
  the mapping the model reads (map_checked) and parses from that mapping; the config and
  tokenizer from the bytes hashed. The docs say what holds after mapping.
- Replay. --check-memory-bytes gives the command line the check's memory a service
  planned in, so every option a transcript records can be given again. The identity
  names the maths library (glibc's version, or macOS's release and build), whose
  exponentials and sines shape the logits. The matmul test now runs the dispatcher the
  projections call (matmul_bf16_auto, one row and three, and the batch).

Tests failing on fbbee6fd9 for the reasons they state:
the_cap_keeps_within_max_new_and_closes_a_reopened_thinking (201 tokens of 200),
every_recorded_option_can_be_given (the flag ignored), an_empty_answer_is_no_program
(an empty program checked). New: a_shard_is_checked_through_the_mapping_the_model_reads,
the_maths_library_is_named.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge wai/quantum-check-exercise (an injected error is corrected from the qubits) into wai/quantum-writer ... 5fbf304a41

The course's tests are fifteen: #645's fourteen and this branch's one (git merged the two
raises of the count as one).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request from a commit 2026-11-06 08:19:31 -05:00

wai-quantum-service: the transcript's maths library in the schema and the claims; the tool snapshot regenerated

dcharlot added 1 commit 2026-11-06 15:25:39 -05:00

Merge main into wai/quantum-writer ... 3d8662628b

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot merged commit 1654254354 into main 2026-11-06 15:26:35 -05:00

dcharlot referenced this pull request from a commit 2026-11-06 15:26:44 -05:00

Merge pull request 'wai-quantum-writer: WAI Quantum writes a program for a task, on its own model' (#646) from wai/quantum-writer into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!646

No description provided.