wai-quantum: a cross-simulator conformance corpus, reproducible with one command, and the surface-code generator's text at p = 0 #662

Merged

dcharlot merged 9 commits from wai/quantum-cross-simulator-corpus into main 2026-11-07 01:27:16 -05:00

Conversation 0 Commits 9 Files changed 173 +64132 -16

2026-11-06

Transaction Science

wai-quantum: the provider template's run reads as a simulator's ...

Main now gives a simulator's result the class `simulated` (the providers
extension, section 8.1); the template's test still held it to `measured`
and failed on the merge.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

90f48f6a75

• View at this point in history

2026-11-06 23:44:50 -05:00

Transaction Science

Merge main into the cross-simulator corpus ...

The workflow keeps both new steps and both sets of paths: the corpus's CI
tier and main's job receipts by two verifiers.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

bdb041e096

• View at this point in history

2026-11-06 23:37:22 -05:00

Transaction Science

wai-quantum: the corpus page gives the full tier's time on both hosts ...

Co-Authored-By: Claude Opus 5.5 <[email protected]>

5f60cfcf89

• View at this point in history

2026-11-06 08:03:57 -05:00

Transaction Science

wai-quantum: the cross-simulator corpus rerun on both hosts, and its page as data ...

Both hosts' full tiers, rerun on the tools of the two commits before this
one (their trees named in each results file):

- macOS arm64 (M3 Ultra): 1638 checks pass, 0 fail; 11 known (the other
  engine's two disagreements, now with their reproducers' cases), 12
  refusals as listed, 11 not run, 12 recorded and not judged.
- Linux x86_64 (Ryzen AI Max+ 395): 1637 pass, 0 fail; 11 known, 12
  refused as listed, 12 not run (the other engine's mirror_n20 timed out
  at 900 s), 12 recorded.
- The self-test catches every planted fault, each on its own check.

Timings rerun on both hosts with each engine's timer stated, its threads,
its ops, and the peak memory of the process that ran it.

The page states results as data: the ranking prose is gone, the account
of the other engine is corrected, repeated_phase_k19's figures are the
modulus and angle the extension states, and the limits say what each
sampled test can and cannot see. report.py prints each reproducer case.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

fffd228757

• View at this point in history

2026-11-06 08:02:39 -05:00

Transaction Science

wai-quantum: the held-or-dropped check decides only where an answer can say ...

The first full run on the M3 Ultra failed two checks on Aer at
below_grid_k60, whose phase of 2π/2^60 moves the state by 2.7e-18, far
below a double's rounding of an amplitude near 1: the share of the step its
answer took was rounding (−21), not a dropped gate. An engine now holds the
gate when its answer is within half the gate's effect of the ideal state,
drops it when within half of the state without it, and otherwise is recorded
as unable to say, not judged. A dropped gate inside the bound still fails
(the drop_phase fault at k = 48 is within 0 of the state without it).

The known disagreement on measure_twice now names the L1 check too, which
the same fault of the other engine fails.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

c5702b44e0

• View at this point in history

2026-11-06 03:51:20 -05:00

Transaction Science

wai-quantum: the cross-simulator corpus's checks can fail where the review found they could not ...

Each change below was shown first as a planted fault that the previous tools let
pass, then as a failure of the check it breaks:

- The engine under test is held to answering. A crash, a time-out or a
  refusal that corpus/expected-refusals.json does not list, in its words,
  fails the run; so does an answer to a circuit it must refuse. The bit-exact
  method's refusals are listed by its reader's rule, computed from the exact
  angles. A time-out kills the engine's whole process group.
- The bit-exact method's op-log, run in double precision, must give the
  circuit's exact state: a phase dropped in lowering now fails, where the
  distance and the referee could not see it. Each small-angle gate is checked
  held or dropped, exactly, on every engine.
- repeated_phase_k19 is held to the state with its phase, beside a quarter
  turn that a wrong sign cannot pass, and to the extension's stated figures.
- No closed-form row is exempt for the bit-exact method. QPE, Grover and the
  phase table are checked in integers, mirror_n10 by the referee.
- Matrix product states are charged what each engine reports it truncated;
  a row whose bound cannot see a wrong phase is recorded, not passed.
- Sampled counts are also held as a whole in L1 distance; the 40-qubit
  Clifford outcomes per outcome, per free bit and in number.
- The account of another engine's errors is corrected, and each claim is a
  case of a reproducer committed beside it.
- Timing records where each engine's timer runs (Aer's own experiment time),
  memory the peak of the process that ran the engine, threads what each was
  given, and ops what the engine ran.
- Provenance names the trees as they were on disk, and the toolchain the
  checkout picks. CI covers every kind of check but the wide pairwise one, on
  referee changes too, and its comment says so.
- The self-test injects every fault and requires its own check to fail, on
  its own engine only.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

3ba49a43ee

• View at this point in history

2026-11-06 03:17:01 -05:00

2026-11-05

Transaction Science

wai-quantum: the cross-simulator corpus's results on two hosts, and its page ...

results/ holds four runs, each naming its host, every version and the git
trees of the engine and tools that produced it:

- conformance on macOS arm64 (M3 Ultra) and on Linux x86_64 (Ryzen AI Max+
  395), the full tier, from the tools as committed in the previous commit:
  1368 and 1363 checks pass, none fails. Two checks on each host are
  recorded as known disagreements of another engine with the exact
  reference, three are refusals by the engine under test of instructions
  its frame simulator does not implement, and seven (nine on Linux) were
  not run, each with its reason.
- run time and peak memory on the same two hosts, five runs a case, with
  the host's load average beside every case.

The page's results section is written from those files by tools/report.py,
and CI holds it to them. The page also says what disagreed and who was
right, what the corpus shows about conventions, where other engines are
faster (gates under many controls, and single precision on one thread),
and the limits of the comparison.

Three changes to the tools since the previous commit, none of which the
conformance run reads: a timing run on the stabilizer adapter measures one
qubit rather than reading the whole group out through the save
instruction's result, which had been most of its time; the memory baseline
uses a two-qubit job, which every adapter accepts; and the page's tables
print whole seconds and MiB, and a refusal's own words.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

3e6ab36d2f

• View at this point in history

2026-11-05 16:52:44 -05:00

Transaction Science

wai-quantum: a cross-simulator conformance corpus, reproducible with one command ...

wai/quantum-cross-simulator/ runs the engine beside pinned third-party
simulators on a published set of circuits and holds every answer to an
exact reference within a tolerance derived from the arithmetic that
produced it.

The corpus (tools/corpus.py, corpus/manifest.json with each circuit's
SHA-256): bit order, GHZ, QFT, Grover, phase estimation, random
Clifford+T, dyadic and arbitrary-angle circuits, mirror circuits, angles
at and below the fixed-point table's resolution, every gate of OpenQASM
3's standard library, measurement in the middle with reset and feedback,
stabilizer circuits to 5000 qubits, noise in closed form, wide circuits by
matrix product states and contraction, and error-correction memory
experiments to distance 15.

The references (tools/exact.py, standard library only): a state vector in
integers at scale 2^-256, exact distributions by walking every branch of a
dynamic program, and an integer stabilizer tableau with the reduced form
of a group. The bit-exact method's op-log is rerun by the existing referee
and must give the same integers and hash.

The tolerances (tools/tolerance.py): the bit-exact method's from the
extension's published bound summed over the op-log it ran; floating point
from a stated per-gate rounding model at each engine's fusion width;
sampled counts by Bernstein's inequality at a family-wise rate of 1e-6.
None is fitted to a result.

reproduce.sh fetches a pinned uv, CPython and every wheel by hash, builds
the engine with --locked, runs every check and prints PASS or FAIL.
--self-test injects six faults and passes only when each fails the run on
its own check. --perf measures run time and peak memory per engine.

provider/seeded_statevector_provider.py wraps a third-party simulator as a
process provider of wai.quantum.provider/1; provider/test.sh holds it to
the protocol's conformance vectors through `providers test`, runs it with
`run --provider` and verifies the receipt.

wai-quantum-cli/examples/cross_simulator.rs is the engine's side: one job
in, one answer out, through the calls the command line and the job service
make.

CI: the Linux leg of wai-quantum-cli.yml runs the CI tier, the self-test,
the provider template and the check that the page's results section is
what the committed results say.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

ec4f2a5acc

• View at this point in history

2026-11-05 15:14:36 -05:00

Transaction Science

wai-quantum: the surface-code memory experiment at p = 0 writes the reference generator's text ...

quantum_frame::surface_code_memory documents that its text matches the
ecosystem's reference generator byte for byte. That held for every p > 0
and not for p = 0: there it wrote each noise instruction with probability
zero (DEPOLARIZE1(0) ..., X_ERROR(0) ...) and the first round's two resets on
two lines, where the reference writes no noise instruction and one reset.
The circuits meant the same thing, so the model and the samples were
unaffected; only the text differed.

It now writes no noise instruction of probability zero, and the two resets
as one instruction. A new fixture test, the reference generator's text at
d = 5 with four rounds and p = 0 (tests/data/memory_z_d5_noiseless.circuit),
failed before the change and passes after it. The fixture count pinned in
wai-quantum-processors.yml and wai-quantum-dsp.yml goes from 2 to 3.

The cross-simulator corpus found it (entry surface_z_d5_noiseless): the
engine's own generator checked against the pinned reference generator.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

112cc14bd3

• View at this point in history

2026-11-05 14:26:00 -05:00