open-standards/wai/quantum-cross-simulator

Cross-simulator conformance corpus

WAI Quantum's engine, run beside other simulators on a published set of circuits, every answer held to a reference within a tolerance derived from the arithmetic that produced it. One command regenerates every result here on a clean machine.

bash wai/quantum-cross-simulator/reproduce.sh --tier full   # the whole corpus: 35 to 50 minutes on the hosts below
bash wai/quantum-cross-simulator/reproduce.sh               # the CI tier, a few minutes
bash wai/quantum-cross-simulator/reproduce.sh --self-test   # every injected fault must be caught
bash wai/quantum-cross-simulator/reproduce.sh --perf --out perf.json

It needs macOS on arm64 or Linux on x86_64, curl, tar, a C compiler and rustup; no account and no key. It fetches the pinned uv release (by SHA-256), CPython 3.12.16 through it, and the engines of env/requirements.lock, every wheel checked against its hash. It builds the engine from this checkout with --locked, runs every check, prints each failure with its measured value and tolerance, and ends with PASS or FAIL (exit 0 or 1). QCORPUS_CACHE says where the downloads go.

What is compared

engine	version	methods compared here
wai-quantum (this repository)	the checkout	bit-exact fixed point (exact), double-precision state vector, stabilizer tableau, matrix product state, tensor network, density matrix, error-correction frames and models, and the whole job path with method auto
Qiskit Aer	0.18.2, with Qiskit 2.6.2	state vector (built in code, and read from the published OpenQASM 3 text), stabilizer, matrix product state, density matrix
Stim	1.16.0	tableau, sampler, detector error models, the reference generator of the error-correction circuits
quimb	1.15.0	dense state, tensor network contracted, matrix product state
MQT DDSIM	2.7.0, with mqt.core 3.11.0	decision-diagram state vector and sampler
qsim	qsimcirq 0.22.1, with Cirq 1.7.0	single-precision state vector and sampler

Each engine runs every job in a process of its own (tools/engines.py). A third-party engine takes each gate by its own name for it where it has one, so a gate it defines differently shows; where it has none, the adapter says so in the result and gives it the gate's matrix. The engine under test is driven through wai-quantum-cli/examples/cross_simulator.rs, which calls what the command line and the job service call.

The corpus

corpus/manifest.json lists every circuit with its parameters, the reason it is there, what it is held to and the SHA-256 of its OpenQASM 3 text; tools/corpus.py writes it, and a run fails if the generator no longer gives those bytes. Programs up to 16 KiB are committed in corpus/qasm/, and the error-correction circuits, which are the pinned reference generator's, in corpus/qec/; a run checks that the generator still writes them. Random circuits draw from SHA-256 in counter mode, and every angle is written exactly: m/2^k of a turn, a rational fraction of a turn, or a decimal number of radians.

family	what it exposes
bit order	which index an X on qubit 0 reads as; named classical bits out of order; a state whose every amplitude differs, so any exchange of qubits shows; an empty register; index bit 19
GHZ, QFT, Grover, phase estimation	closed forms: two amplitudes of 1/√2; e^{2πixy/2^n}/2^(n/2) with every controlled phase to 2π/2^n; sin²((2k+1)θ); an outcome of probability 1, and for a phase of 1/3 the closed law, which the bit-exact method must refuse to round
random Clifford+T, dyadic, arbitrary angles	universal circuits in the table's gates, rotations that become many table phases in fixed point, and decimal angles no dyadic grid holds
mirror	U then U†, at widths past the integer reference: the answer is |0…0⟩ exactly
small angles	phases of 2π/2^k for k up to 32, the table's last entry, and below it to k = 60, each with a check of whether the engine held the phase or dropped it; p(2π/2^19) applied 2^18 times (a half turn, whose fixed-point result the extension states) and 2^17 times (a quarter turn), each held to the state with its phase
gate conventions	every gate of OpenQASM 3's standard library, and modifiers, on a state with no symmetry; rotations by whole turns, whose sign is the convention; a controlled global phase
measurement in the middle	teleportation, reset of an entangled qubit, measuring twice, feedback, a GHZ state by fusion, repeat-until-success: exact laws from every branch
stabilizer circuits	GHZ and cluster states to 5000 qubits; random Clifford layers at 63, 64, 65, 127, 128 and 129 qubits (either side of a machine word) and to 2000; a Clifford circuit as a state vector and a tableau; outcomes at 40 qubits
noise	depolarizing channels after X and cx chains, whose expectation values are closed forms
wide circuits	GHZ to 100 qubits and the QFT at 30 and 40 by matrix product states and contraction, and at 30 at a bond far above what it reaches; a shallow brickwork at 26; a 40-qubit mirror
error correction	the rotated surface-code memory experiment in Z (d = 3 … 15) and X, the unrotated layout, repetition codes to d = 25, a noiseless run, a colour code whose cycle turns the axes with C_XYZ, and hand-written circuits for inverted targets, tags, flip noise, merging, Pauli observables, a correlated error, a general Pauli channel, and two circuits that must be refused

How a result is judged

The references are exact or as good as exact, and share no code with any engine (tools/exact.py, standard library only):

• A state vector in integers. Amplitudes are integers at scale 2^-256; sine and cosine come from an integer π and an integer series. Every run checks that it gives the extension's phase table, P(1) … P(32) rounded to the nearest 2^-30, entry for entry against the referee's, and that its states meet the closed forms within 2^40 units of 2^-256: the QFT's amplitudes, GHZ's, Grover's sin²((2k+1)θ) (computed in integers), phase estimation's outcome of probability 1, and its law for a phase of 1/3.
• Exact distributions. A program that measures in the middle is walked branch by branch.
• An integer stabilizer tableau, and the reduced row echelon form of a group with its signs, so that two descriptions of a stabilizer state compare as bytes.
• The referee (../quantum-conformance/tools/referee.py) reruns the bit-exact method's op-log in its own integer arithmetic and must give the engine's integers and hash, on every circuit of up to 14 qubits the method runs. Beside it, the op-log run in double precision must give the circuit's exact state, modulo a global phase, within that run's rounding and the distance the reader's rule moves angles: so the integers are tied to the op-log, and the op-log to the circuit.

Each tolerance comes from tools/tolerance.py, and none is fitted to a result:

• The bit-exact method: the extension's published bound (quantum-sim.md §5, at most 1 + √2/2 units of 2^-30 per component per inexact gate), summed over the op-log the engine ran.
• Floating point: a gate of width w costs C(w)·u in the Euclidean norm, C(w) = (2√2 + (2^w − 1)
  • 2 + π)·2^(w/2), from the complex products, the row sum, the matrix entries and the angle's rounding; u = 2^-53 in double and 2^-24 in single precision; widths are each engine's fusion. Contractions, decision diagrams (whose merging tolerance, 2^-42, is read from mqt.core's header) and matrix product states (with LAPACK's unstated SVD constant taken as 16·m, and the weight their truncation can drop) are stated in the same file.
• Matrix product states: the rounding of each two-site update's SVD (a gate on qubits d apart takes 2d − 1 of them, its routing swaps included), and 2√w for each truncation of weight w, as each engine reports what it dropped: the engine under test its summed weight over its truncations, Aer every truncation from its log, and quimb run with an absolute cutoff whose most per update is known. A row whose bound is not below half the amplitudes it checks is recorded and not judged.
• Sampled counts: every outcome's count against its exact probability by Bernstein's inequality, and the whole distribution against the exact one in L1 distance, twice the total variation (the bound of Weissman and others, 2003, which holds for any number of outcomes); an outcome of probability 0 that appears at all fails. A Clifford circuit's outcomes at 40 qubits: every shot inside the affine space its stabilizers fix, no outcome more often than its probability allows, and each bit the space leaves free reading 1 half the time.
• Error-correction models: the same symptom sets, probabilities within 16u per noise site of the circuit; detection rates against the rates the model gives, by Bernstein's inequality.
• Every sampled test of a run (counts, stabilizer outcomes, detection rates) takes a share of one family-wise error rate of 10^-6, so a correct engine fails a run on sampling with probability at most 10^-6.

A state is held to its reference modulo a global phase, which no measurement sees; the distance with the phase is recorded beside it (strict). Where the phase is the result (the repeated phases), the verdict is the distance with it. The bounds are worst cases and so are loose: each result records how far inside its bound an engine is, and the table below gives the largest such ratio. Below an engine's bound a distance cannot tell a small-angle gate from its absence, so each small-angle circuit also asks of every engine whether it held the gate's phase or dropped it, exactly, in integers.

The engine under test is held to answering. A crash, a time-out, or a refusal that corpus/expected-refusals.json does not list for that circuit in those words fails the run; an answer to a circuit that file says it must refuse fails it too. The file lists the bit-exact method's refusals by its reader's rule (an angle no grid of 2^32 holds within 10^-14 of a turn), computed from the circuits' exact angles, and four error-correction circuits. Another engine's refusal or crash is recorded as not run, with its words.

A run can be seen to fail. --self-test runs one circuit for each fault compare.py can inject, and passes only if each fails the run on the check it breaks, on the engine it was put into, and on no other engine:

• an amplitude moved by 10^-9; the qubits of a state reversed; one fixed-point bit flipped; a stabilizer's sign flipped; a mechanism's probability raised by one part in 10^6;
• 5% of an outcome's shots moved to another that can occur (caught by the per-outcome test, on a distribution with variance); every outcome's count moved by a tenth of itself, up and down in turn (caught only by the L1 distance); every shot of a 40-qubit Clifford sampler given to one outcome; a detector's count raised by a quarter; a sampler's answer with no observable;
• a phase dropped from the bit-exact op-log, its integers and hash made again to match (caught only by the op-log against the circuit); a small-angle gate dropped from a double-precision state, inside its bound (caught only by the held-or-dropped check); a half turn on a repeated phase (caught only by the distance with the phase); a global phase on the bit-exact state (caught by the closed-form amplitudes, which it was once exempt from); a phase of 0.5 rad on a matrix-product amplitude (caught by a bound charged what that engine truncated);
• the engine under test refusing, crashing on a job or on a model, hanging, taking an angle it must refuse, or refusing in words no list holds, or in other words than its list's.

Results

The committed runs, one file per host, are in results/: every check with its measured value and tolerance, the host and every version. Each file names the git trees of the engine and of the tools that produced it, as they were on disk (git rev-parse <commit>:wai/wai-quantum gives the same id for any commit holding the same engine). A cell below is checks passed of those made, with the largest ratio of measured value to tolerance among the passes in brackets.

What agreed

On both hosts, every engine met its reference within its tolerance on every circuit it ran, with the exceptions listed next. That covers eight methods of the engine under test and eleven of the others, on states, sampled counts, stabilizer groups to 5000 qubits, amplitudes to 100 qubits, noisy expectation values, and detector error models and detection rates to distance 15. The engine under test answered every circuit, and refused only those corpus/expected-refusals.json lists, in the words it lists.

The bit-exact method's integers were reproduced bit for bit by the referee on every circuit of up to 14 qubits it ran; its op-log, run in double precision, gave each circuit's exact state; and its distance from the ideal state stayed inside the extension's published bound everywhere. On repeated_phase_k19, p(2π/2^19) applied 2^18 times to |1⟩, its amplitude ends with modulus 1.0000188705 and 8.99·10^-6 rad from the half turn (a distance of 2.09·10^-5 from −|1⟩), where the extension states modulus 1.0000189 and within 10^-5 rad. These are integers, the same on both hosts.

Disagreements with the references

• The engine under test, wrong; fixed. quantum_frame::surface_code_memory documents that its text is the reference generator's, byte for byte. At p = 0 it was not: it wrote every noise instruction with probability 0 and the first round's resets on two lines, where the reference writes no noise instruction and one reset line (surface_z_d5_noiseless). The two circuits meant the same, so models and samples agreed; the text did not. Fixed in the engine, with a test against the reference's text (wai-quantum/tests/frame_reference.rs).
• Another engine. ddsim.statevector on measure_twice (h, then the same qubit measured into two bits) returns 00 and 10, where a repeated measurement repeats: 00 and 11. On repeat_until_success it raises an error and gives no counts. Each claim about it, with the smaller programs that show what the error needs, is a case of a reproducer committed beside corpus/known-disagreements.json (corpus/reproducers/), which every run that holds the circuit reruns; the two checks are reported as known. Only other engines may be listed there.
• The reference, wrong once. While this corpus was written, the integer reference's ryy used the wrong fixed rotation. It showed as every engine agreeing with every other and none with the reference, and was corrected before any result here was recorded. A reference is examined first when that pattern appears.

Conventions the corpus shows

• Global phase. Every engine keeps OpenQASM 3's global phases except the bit-exact method, which lowers rz, rx, ry and U through phase gates and so differs from the standard's matrices by a global phase (a strict distance of up to 1.97 on a rotation by a whole turn). No measurement sees it, and under a control the method is right (controlled_rz_full_turn, gate_crz, gate_cu). A result on this method states the phase (accuracy.global_phase, exactly, in 2^33rds of a turn), and the corpus holds the method's state to the reference's with that phase, nothing fitted.
• Angles below the table. The bit-exact method reads an angle within 10^-14 of a turn of a dyadic one as that dyadic. A phase of 2π/2^k is therefore refused for k = 33 … 46 ([Q500]), and for k ≥ 47 runs as no phase at all, labelled bit-exact: its op-log is the circuit's with the angle moved by 2^-k of a turn, which the op-log check allows by the reader's rule, and its state is within 2.3·10^-14 of the ideal one, far inside the method's resolution. The result and the plan say so (accuracy.below_table, with how many and the largest), and an angle moved to a dyadic neighbour other than zero is stated too (accuracy.moved_angles). Method auto sends k = 33 … 46 to double precision.
• Held or dropped. Every double- and single-precision engine here held the phase at every k where its answer can say. The decision-diagram engine dropped it at k = 47, 48, 52 and 60, each an angle below its parameter tolerance of 10^-13 rad, and held it at k ≤ 40. At k = 60 the phase moves the state by 2.7·10^-18, less than half of most answers' own rounding: those are recorded as unable to say, and the single-precision engine's answer, whose rounding there was smaller, held it.
• Bit order. Every engine's answers were brought to one convention (qubit 0 the lowest bit of an index, classical bit 0 the last character of an outcome) by its adapter, and the bit-order family holds each adapter to it; the self-test's reversed state fails there.

Run time and memory

The performance tables below are one shared host each, on one day; read the Limits before any figure. Each row is one engine on one case, as data:

• threads: what the engine was given. all (32) is every CPU the process may run on, which the engine under test's state vector and tensor network use; 1 (no threads) a method with no threads of its own (the engine under test's bit-exact, stabilizer, matrix-product-state and error-correction methods); a number, the count passed in the engine's own setting; not set, an engine the adapter gives no thread setting; 1 (pinned), the process held to one CPU (Linux only).
• ops: the operations the engine ran, where it reports them: the driver's count for the engine under test, and the gates of Aer's circuit after its conversion.
• seconds: the engine's own call, the median of the runs with the minimum in brackets. The list above each table says where each engine's timer starts and stops.
• peak MiB: the peak resident set of the process that ran the engine (the driver for the engine under test; for another engine, its adapter, which holds the interpreter and its packages), as the system counts it for that process alone; the baseline beside it is the same measure given a two-qubit job.

The last table runs the same harness on the engine under test built before and after a gate under many controls stood whole on its state vector, one build straight after the other on one host; ops gives the operations each build ran. wai.exact_whole is the bit-exact method asked to hold such a gate whole ("options": {"controls": "whole"}), which gives other bits than its default; wai.exact, its default, runs standard gates and gives the bits it gave before. Past 16 controls the standard form refuses ([Q208]), so grover_n20 has no "before".

conformance-linux-x86_64.json

Host: AMD RYZEN AI MAX+ 395 w/ Radeon 8060S, Ubuntu 26.04.1 LTS, 32 logical CPUs, 123 GiB, x86_64. Tier full, 2911.3 s. Checks: info 12, known 11, not run 14, pass 1738, refused 10.

family	wai.exact	wai.statevector	wai.stabilizer	wai.auto	wai.mps	wai.tensor_network	wai.density_matrix	wai.qec	aer.statevector	aer.qasm_import	aer.stabilizer	aer.mps	aer.density_matrix	stim.tableau	quimb.dense	quimb.tensor_network	quimb.mps	ddsim.statevector	qsim.statevector	pairs and references	references
bit_order	24/24 (0.068)	8/8 (0.015)	12/12	8/8 (0.0069)					8/8 (0.016)	6/6 (0.016)	5/5			7/7	6/6 (0.019)	6/6 (0.031)		8/8 (0.00015)	8/8 (0.013)	15/15 (0.0084)	5/5
clifford_large	4/4 (0.012)	1/1 (0.00077)	34/34 (0.43)	1/1 (7.8e-05)					1/1 (0.00084)	1/1 (0.00084)	16/16 (0.51)			19/19 (0.3)	1/1 (0.0017)	1/1 (4.5e-08)		1/1 (2.4e-07)	1/1 (0.00052)		15/15
dynamic		12/12 (0.29)	8/8 (0.21)	12/12 (0.29)					12/12 (0.36)					8/8 (0.29)				8/8 (0.36) · 11 known	6 not run
gate_conventions	188/188 (0.069)	47/47 (0.017)		47/47 (0.066)					47/47 (0.018)	47/47 (0.018)					46/46 (0.017) · 1 not run	46/46 (0.032) · 1 not run		47/47 (0.00015)	46/46 (0.026) · 1 not run
ghz	20/20 (0.042)	7/7 (0.21)	2/2 (0.022)	7/7 (0.022)					7/7 (0.059)	5/5 (0.011)				2/2 (0.29)	5/5 (0.0094)	5/5 (0.041)		7/7 (0.082)	7/7 (0.14)		5/5
grover	16/16 (0.029)	6/6 (0.12)		6/6 (0.14)					6/6 (0.25)	4/4 (0.0043)					4/4 (0.0079)	4/4 (0.00034)		6/6 (0.21)	6/6 (0.27)		4/4
mirror	7/7 (0.007)	3/3 (0.00036)							3/3 (0.00068)						3/3 (0.00036)			2/2 (7e-05) · 1 not run	3/3 (0.00057)
noise							3/3 (0.00093)						3/3 (0.00093)
other																					3/3
qec								29/29 (0.61) · 2 refused as listed						22/22 (0.53) · 2 not run						44/44 (0.016)
qft	25/25 (0.042)	9/9 (0.58)		8/8 (0.66)					9/9 (0.61)	6/6 (0.009)					7/7 (0.0062)	5/5 (0.0057)		9/9 (0.63)	9/9 (0.63)	15/15 (0.0027)	5/5
qpe	12/12 (0.017) · 2 refused as listed	5/5 (0.0091)		5/5 (0.013)					5/5 (0.0058)	5/5 (0.0075)					5/5 (0.0071)	4/4 (0.0014)		5/5 (4.8e-06)	5/5 (0.0034)		5/5
random_clifford_t	12/12 (0.039)	3/3 (0.0047)		3/3 (0.017)					3/3 (0.0042)	3/3 (0.0042)					3/3 (0.003)	3/3 (0.00024)		3/3 (3e-05)	3/3 (0.001)
random_dyadic	8/8 (0.0067)	2/2 (0.0018)		2/2 (0.0016)					2/2 (0.0027)	2/2 (0.0036)					2/2 (0.0019)	2/2 (0.00015)		2/2 (6.8e-06)	2/2 (0.004)
random_rotations	3 refused as listed	5/5 (0.48)		5/5 (0.48)					5/5 (0.5)	3/3 (0.0034)					3/3 (0.0018)	3/3 (0.00014)		5/5 (0.4) · 1 not run	5/5 (0.4)	6/6 (0.0017)
small_angle	62/62 (0.9) · 3 refused as listed	32/32 (0.016)	2/2	18/18 (0.098)					32/32 (0.018)	29/29 (0.016)	1/1			1/1	30/30 (0.019)	29/29 (0.016)		29/29 (0.15)	33/33 (0.0075)		1/1
wide_mps					6/6 (0.00017)	5/5 (0.00057)						4/4 (0.00023) · 1 not run				5/5 (0.043)	5/5 (7.3e-05)
wide_tn					1/1 (0.00093)	1/1 (6.4e-07)						1/1 (0.0021)				1/1 (0.02)	1/1 (0.0011)			21/21 (0.02)

Known disagreements of other engines with the exact reference (corpus/known-disagreements.json):

• repeat_until_success, ddsim.statevector: ValueError: [changePermutation] Target must be an injective mapping using only keys and logical qubits from the source permutation. | known: MQT DDSIM 2.7.0's qasm_simulator raises ValueError [changePermutation] on this program (two qubits; qubit 1 measured three times in the middle, with gates between and an x conditioned on each reading; four classical bits) and gives no counts. It raises the same without the conditioned x, and on qubit 1 measured, h, measured again, then qubit 0 measured into a third bit; with no third bit it gives counts. Each is a case of the reproducer.
• measure_twice, ddsim.statevector: 10: 50067 shots, probability 0; 11: 0 shots, expected 50000.0 ± 1084.8 | known: MQT DDSIM 2.7.0's qasm_simulator, given h q[0]; c[0] = measure q[0]; c[1] = measure q[0]; with nothing between the two measurements, returns 00 and 10: c[0] is never set. With x in place of h it gives 10 on every shot; with a barrier between the measurements, 00 and 11; with c[1] measured first and c[0] second, ValueError [changePermutation]. Each is a case of the reproducer, with the circuit built in Qiskit 2.6.2.
• measure_twice, ddsim.statevector: known: MQT DDSIM 2.7.0's qasm_simulator, given h q[0]; c[0] = measure q[0]; c[1] = measure q[0]; with nothing between the two measurements, returns 00 and 10: c[0] is never set. With x in place of h it gives 10 on every shot; with a barrier between the measurements, 00 and 11; with c[1] measured first and c[0] second, ValueError [changePermutation]. Each is a case of the reproducer, with the circuit built in Qiskit 2.6.2.
• measure_twice, ddsim.statevector, reproducer (corpus/reproducers/ddsim_repeated_measurement.py): h, then c[0] and c[1] measured from the qubit with nothing between: only 00 and 10 come — saw ['00', '10']
• measure_twice, ddsim.statevector, reproducer (corpus/reproducers/ddsim_repeated_measurement.py): x, then the same two measurements: 10 on every shot — saw ['10']
• measure_twice, ddsim.statevector, reproducer (corpus/reproducers/ddsim_repeated_measurement.py): h, the two measurements with a barrier between: 00 and 11 — saw ['00', '11']
• measure_twice, ddsim.statevector, reproducer (corpus/reproducers/ddsim_repeated_measurement.py): h, then c[1] measured first and c[0] second: an error naming changePermutation — saw ValueError: [changePermutation] Target must be an injective mapping using only keys and logical qubits from the source permutation.
• repeat_until_success, ddsim.statevector, reproducer (corpus/reproducers/ddsim_feedback_rounds.py): repeat_until_success: an error naming changePermutation, and no counts — saw ValueError: [changePermutation] Target must be an injective mapping using only keys and logical qubits from the source permutation.
• repeat_until_success, ddsim.statevector, reproducer (corpus/reproducers/ddsim_feedback_rounds.py): the same three rounds without the conditioned x: the same error — saw ValueError: [changePermutation] Target must be an injective mapping using only keys and logical qubits from the source permutation.
• repeat_until_success, ddsim.statevector, reproducer (corpus/reproducers/ddsim_feedback_rounds.py): qubit 1 measured, h, measured again, then qubit 0 measured into a third bit: the same error — saw ValueError: [changePermutation] Target must be an injective mapping using only keys and logical qubits from the source permutation.
• repeat_until_success, ddsim.statevector, reproducer (corpus/reproducers/ddsim_feedback_rounds.py): the same without qubit 0's measurement into a third bit: counts — saw counts over 4 outcomes

Refusals by the engine under test, each listed in corpus/expected-refusals.json with its words (10)

• qpe_t6_third, wai.exact: the angle 2.0943951023931953 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• qpe_t10_third, wai.exact: the angle 2.0943951023931953 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• rotations_n6, wai.exact: the angle 1.6459015101789476 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• rotations_n10, wai.exact: the angle 0.49840613019150304 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• rotations_n12, wai.exact: the angle 1.437905822610216 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• below_grid_k33, wai.exact: the angle 0.0000000007314590396335798 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• below_grid_k40, wai.exact: the angle 0.000000000005714523747137342 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• tiny_radians, wai.exact: the angle 0.000000000001 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• qec_refused_channel_with_no_independent_form, wai.qec: line 2: PAULI_CHANNEL_1(0.1, 0.1, 0): its exclusive outcomes have no form as independent X, Y and Z flips, which this simulator and a detector error model both need
• qec_refused_random_detector, wai.qec: a detector or observable is not deterministic (it anticommutes with op 0)

Checks other engines did not make, and why (14)

• rotations_wide_n20, ddsim.statevector: did not finish within 900 s
• mirror_n20, ddsim.statevector: did not finish within 900 s
• gate_gphase_controlled, quimb.dense: no gate gphase
• gate_gphase_controlled, quimb.tensor_network: no gate gphase
• gate_gphase_controlled, qsim.statevector: a global phase gate
• teleport, qsim.statevector: measurement in the middle (or one qubit measured twice)
• reset_after_superposition, qsim.statevector: measurement in the middle (or one qubit measured twice)
• measure_twice, qsim.statevector: measurement in the middle (or one qubit measured twice)
• feedback_flip, qsim.statevector: measurement in the middle (or one qubit measured twice)
• ghz_by_fusion, qsim.statevector: measurement in the middle (or one qubit measured twice)
• repeat_until_success, qsim.statevector: measurement in the middle (or one qubit measured twice)
• mps_ghz_n100, aer.mps: CircuitTooWideForTarget: 'Number of qubits (100) in circuit-46 is greater than maximum (63) in the coupling_map'
• qec_refused_channel_with_no_independent_form, stim.tableau: ValueError: Encountered the operation PAULI_CHANNEL_1 during error analysis, but this operation requires the approximate_disjoint_errors option to be enabled. If you're calling from python, using stim.Circuit.detector_error_model, you need to add the argument approximate_disjoint_errors=True. If
• qec_refused_random_detector, stim.tableau: ValueError: The circuit contains non-deterministic detectors. To make an SVG picture of the problem, you can use the python API like this: your_circuit.diagram('detslice-with-ops-svg', tick=range(0, 5), filter_coords=['D0', ]) or the command line API like this: stim diagram --in your_circui

Recorded and not judged (9)

• below_grid_k47, ddsim.statevector: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured -4.3e-78. drops it: the angle, 4.46e-14 rad, is below its parameter tolerance of 1e-13 rad, where it may read an angle as 0
• below_grid_k48, ddsim.statevector: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured -4.3e-78. drops it: the angle, 2.23e-14 rad, is below its parameter tolerance of 1e-13 rad, where it may read an angle as 0
• below_grid_k52, ddsim.statevector: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured -4.3e-78. drops it: the angle, 1.4e-15 rad, is below its parameter tolerance of 1e-13 rad, where it may read an angle as 0
• below_grid_k60, wai.statevector: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured 1. neither can be told: its distance from the ideal state, 2.22e-16, is more than half the gate's effect, 2.72e-18
• below_grid_k60, aer.statevector: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured -21. neither can be told: its distance from the ideal state, 6.12e-17, is more than half the gate's effect, 2.72e-18
• below_grid_k60, aer.qasm_import: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured -21. neither can be told: its distance from the ideal state, 6.12e-17, is more than half the gate's effect, 2.72e-18
• below_grid_k60, quimb.dense: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured 1. neither can be told: its distance from the ideal state, 2.22e-16, is more than half the gate's effect, 2.72e-18
• below_grid_k60, quimb.tensor_network: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured 1. neither can be told: its distance from the ideal state, 1.11e-16, is more than half the gate's effect, 2.72e-18
• below_grid_k60, ddsim.statevector: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured -4.3e-78. drops it: the angle, 5.45e-18 rad, is below its parameter tolerance of 1e-13 rad, where it may read an angle as 0

Versions: cirq-core 1.7.0, mqt.core 3.11.0, mqt.ddsim 2.7.0, numpy 2.5.4, python 3.12.16, qiskit 2.6.2, qiskit-aer 0.18.2, qsimcirq 0.22.1, quimb 1.15.0, scipy 1.18.1, stim 1.16.0.

conformance-macos-arm64.json

Host: Apple M3 Ultra, macOS 27.2, 32 logical CPUs, 256 GiB, arm64. Tier full, 2092.6 s. Checks: info 12, known 11, not run 12, pass 1743, refused 10.

family	wai.exact	wai.statevector	wai.stabilizer	wai.auto	wai.mps	wai.tensor_network	wai.density_matrix	wai.qec	aer.statevector	aer.qasm_import	aer.stabilizer	aer.mps	aer.density_matrix	stim.tableau	quimb.dense	quimb.tensor_network	quimb.mps	ddsim.statevector	qsim.statevector	pairs and references	references
bit_order	24/24 (0.068)	8/8 (0.015)	12/12	8/8 (0.0069)					8/8 (0.016)	6/6 (0.016)	5/5			7/7	6/6 (0.019)	6/6 (0.031)		8/8 (0.00015)	8/8 (0.013)	15/15 (0.0084)	5/5
clifford_large	4/4 (0.012)	1/1 (0.00077)	34/34 (0.43)	1/1 (7.8e-05)					1/1 (0.00084)	1/1 (0.00084)	16/16 (0.51)			19/19 (0.34)	1/1 (0.0017)	1/1 (5e-08)		1/1 (2.4e-07)	1/1 (0.0024)		15/15
dynamic		12/12 (0.29)	8/8 (0.21)	12/12 (0.29)					12/12 (0.36)					8/8 (0.32)				8/8 (0.25) · 11 known	6 not run
gate_conventions	188/188 (0.069)	47/47 (0.017)		47/47 (0.066)					47/47 (0.018)	47/47 (0.018)					46/46 (0.017) · 1 not run	46/46 (0.032) · 1 not run		47/47 (0.00015)	46/46 (0.026) · 1 not run
ghz	20/20 (0.042)	7/7 (0.21)	2/2 (0.022)	7/7 (0.022)					7/7 (0.059)	5/5 (0.011)				2/2 (0.29)	5/5 (0.0094)	5/5 (0.041)		7/7 (0.21)	7/7 (0.14)		5/5
grover	16/16 (0.029)	6/6 (0.12)		6/6 (0.14)					6/6 (0.25)	4/4 (0.0043)					4/4 (0.0079)	4/4 (0.0003)		6/6 (0.21)	6/6 (0.27)		4/4
mirror	7/7 (0.007)	3/3 (0.00036)							3/3 (0.0011)						3/3 (0.00049)			3/3 (0.00079)	3/3 (0.00059)
noise							3/3 (0.00093)						3/3 (0.00093)
other																					3/3
qec								29/29 (0.61) · 2 refused as listed						22/22 (0.52) · 2 not run						44/44 (0.016)
qft	25/25 (0.042)	9/9 (0.58)		8/8 (0.66)					9/9 (0.61)	6/6 (0.009)					7/7 (0.0062)	5/5 (0.0093)		9/9 (0.67)	9/9 (0.63)	15/15 (0.0027)	5/5
qpe	12/12 (0.017) · 2 refused as listed	5/5 (0.0091)		5/5 (0.013)					5/5 (0.005)	5/5 (0.0074)					5/5 (0.0068)	4/4 (0.0011)		5/5 (7e-06)	5/5 (0.0037)		5/5
random_clifford_t	12/12 (0.039)	3/3 (0.0047)		3/3 (0.017)					3/3 (0.0042)	3/3 (0.0042)					3/3 (0.003)	3/3 (0.00021)		3/3 (3.7e-05)	3/3 (0.0012)
random_dyadic	8/8 (0.0067)	2/2 (0.0018)		2/2 (0.0016)					2/2 (0.0031)	2/2 (0.0038)					2/2 (0.0014)	2/2 (0.00016)		2/2 (8e-06)	2/2 (0.004)
random_rotations	3 refused as listed	5/5 (0.48)		5/5 (0.48)					5/5 (0.5)	3/3 (0.0039)					3/3 (0.0019)	3/3 (0.00014)		5/5 (0.49)	5/5 (0.4)	10/10 (0.067)
small_angle	62/62 (0.9) · 3 refused as listed	32/32 (0.016)	2/2	18/18 (0.098)					32/32 (0.018)	29/29 (0.016)	1/1			1/1	30/30 (0.019)	29/29 (0.016)		29/29 (0.15)	33/33 (0.0075)		1/1
wide_mps					6/6 (0.00017)	5/5 (0.00057)						4/4 (0.00023) · 1 not run				5/5 (0.043)	5/5 (7.3e-05)
wide_tn					1/1 (0.00093)	1/1 (6.4e-07)						1/1 (0.0033)				1/1 (0.019)	1/1 (0.0006)			21/21 (0.015)

Known disagreements of other engines with the exact reference (corpus/known-disagreements.json):

• repeat_until_success, ddsim.statevector: ValueError: [changePermutation] Target must be an injective mapping using only keys and logical qubits from the source permutation. | known: MQT DDSIM 2.7.0's qasm_simulator raises ValueError [changePermutation] on this program (two qubits; qubit 1 measured three times in the middle, with gates between and an x conditioned on each reading; four classical bits) and gives no counts. It raises the same without the conditioned x, and on qubit 1 measured, h, measured again, then qubit 0 measured into a third bit; with no third bit it gives counts. Each is a case of the reproducer.
• measure_twice, ddsim.statevector: 10: 49689 shots, probability 0; 11: 0 shots, expected 50000.0 ± 1084.8 | known: MQT DDSIM 2.7.0's qasm_simulator, given h q[0]; c[0] = measure q[0]; c[1] = measure q[0]; with nothing between the two measurements, returns 00 and 10: c[0] is never set. With x in place of h it gives 10 on every shot; with a barrier between the measurements, 00 and 11; with c[1] measured first and c[0] second, ValueError [changePermutation]. Each is a case of the reproducer, with the circuit built in Qiskit 2.6.2.
• measure_twice, ddsim.statevector: known: MQT DDSIM 2.7.0's qasm_simulator, given h q[0]; c[0] = measure q[0]; c[1] = measure q[0]; with nothing between the two measurements, returns 00 and 10: c[0] is never set. With x in place of h it gives 10 on every shot; with a barrier between the measurements, 00 and 11; with c[1] measured first and c[0] second, ValueError [changePermutation]. Each is a case of the reproducer, with the circuit built in Qiskit 2.6.2.
• measure_twice, ddsim.statevector, reproducer (corpus/reproducers/ddsim_repeated_measurement.py): h, then c[0] and c[1] measured from the qubit with nothing between: only 00 and 10 come — saw ['00', '10']
• measure_twice, ddsim.statevector, reproducer (corpus/reproducers/ddsim_repeated_measurement.py): x, then the same two measurements: 10 on every shot — saw ['10']
• measure_twice, ddsim.statevector, reproducer (corpus/reproducers/ddsim_repeated_measurement.py): h, the two measurements with a barrier between: 00 and 11 — saw ['00', '11']
• measure_twice, ddsim.statevector, reproducer (corpus/reproducers/ddsim_repeated_measurement.py): h, then c[1] measured first and c[0] second: an error naming changePermutation — saw ValueError: [changePermutation] Target must be an injective mapping using only keys and logical qubits from the source permutation.
• repeat_until_success, ddsim.statevector, reproducer (corpus/reproducers/ddsim_feedback_rounds.py): repeat_until_success: an error naming changePermutation, and no counts — saw ValueError: [changePermutation] Target must be an injective mapping using only keys and logical qubits from the source permutation.
• repeat_until_success, ddsim.statevector, reproducer (corpus/reproducers/ddsim_feedback_rounds.py): the same three rounds without the conditioned x: the same error — saw ValueError: [changePermutation] Target must be an injective mapping using only keys and logical qubits from the source permutation.
• repeat_until_success, ddsim.statevector, reproducer (corpus/reproducers/ddsim_feedback_rounds.py): qubit 1 measured, h, measured again, then qubit 0 measured into a third bit: the same error — saw ValueError: [changePermutation] Target must be an injective mapping using only keys and logical qubits from the source permutation.
• repeat_until_success, ddsim.statevector, reproducer (corpus/reproducers/ddsim_feedback_rounds.py): the same without qubit 0's measurement into a third bit: counts — saw counts over 4 outcomes

Refusals by the engine under test, each listed in corpus/expected-refusals.json with its words (10)

• qpe_t6_third, wai.exact: the angle 2.0943951023931953 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• qpe_t10_third, wai.exact: the angle 2.0943951023931953 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• rotations_n6, wai.exact: the angle 1.6459015101789476 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• rotations_n10, wai.exact: the angle 0.49840613019150304 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• rotations_n12, wai.exact: the angle 1.437905822610216 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• below_grid_k33, wai.exact: the angle 0.0000000007314590396335798 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• below_grid_k40, wai.exact: the angle 0.000000000005714523747137342 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• tiny_radians, wai.exact: the angle 0.000000000001 is not a dyadic multiple of 2π (2πm/2^k, k ≤ 32) [Q500]
• qec_refused_channel_with_no_independent_form, wai.qec: line 2: PAULI_CHANNEL_1(0.1, 0.1, 0): its exclusive outcomes have no form as independent X, Y and Z flips, which this simulator and a detector error model both need
• qec_refused_random_detector, wai.qec: a detector or observable is not deterministic (it anticommutes with op 0)

Checks other engines did not make, and why (12)

• gate_gphase_controlled, quimb.dense: no gate gphase
• gate_gphase_controlled, quimb.tensor_network: no gate gphase
• gate_gphase_controlled, qsim.statevector: a global phase gate
• teleport, qsim.statevector: measurement in the middle (or one qubit measured twice)
• reset_after_superposition, qsim.statevector: measurement in the middle (or one qubit measured twice)
• measure_twice, qsim.statevector: measurement in the middle (or one qubit measured twice)
• feedback_flip, qsim.statevector: measurement in the middle (or one qubit measured twice)
• ghz_by_fusion, qsim.statevector: measurement in the middle (or one qubit measured twice)
• repeat_until_success, qsim.statevector: measurement in the middle (or one qubit measured twice)
• mps_ghz_n100, aer.mps: CircuitTooWideForTarget: 'Number of qubits (100) in circuit-46 is greater than maximum (63) in the coupling_map'
• qec_refused_channel_with_no_independent_form, stim.tableau: ValueError: Encountered the operation PAULI_CHANNEL_1 during error analysis, but this operation requires the approximate_disjoint_errors option to be enabled. If you're calling from python, using stim.Circuit.detector_error_model, you need to add the argument approximate_disjoint_errors=True. If
• qec_refused_random_detector, stim.tableau: ValueError: The circuit contains non-deterministic detectors. To make an SVG picture of the problem, you can use the python API like this: your_circuit.diagram('detslice-with-ops-svg', tick=range(0, 5), filter_coords=['D0', ]) or the command line API like this: stim diagram --in your_circui

Recorded and not judged (9)

• below_grid_k47, ddsim.statevector: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured -4.3e-78. drops it: the angle, 4.46e-14 rad, is below its parameter tolerance of 1e-13 rad, where it may read an angle as 0
• below_grid_k48, ddsim.statevector: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured -4.3e-78. drops it: the angle, 2.23e-14 rad, is below its parameter tolerance of 1e-13 rad, where it may read an angle as 0
• below_grid_k52, ddsim.statevector: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured -4.3e-78. drops it: the angle, 1.4e-15 rad, is below its parameter tolerance of 1e-13 rad, where it may read an angle as 0
• below_grid_k60, wai.statevector: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured 1. neither can be told: its distance from the ideal state, 2.22e-16, is more than half the gate's effect, 2.72e-18
• below_grid_k60, aer.statevector: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured -21. neither can be told: its distance from the ideal state, 6.12e-17, is more than half the gate's effect, 2.72e-18
• below_grid_k60, aer.qasm_import: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured -21. neither can be told: its distance from the ideal state, 6.12e-17, is more than half the gate's effect, 2.72e-18
• below_grid_k60, quimb.dense: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured 1. neither can be told: its distance from the ideal state, 2.22e-16, is more than half the gate's effect, 2.72e-18
• below_grid_k60, quimb.tensor_network: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured 1. neither can be told: its distance from the ideal state, 1.11e-16, is more than half the gate's effect, 2.72e-18
• below_grid_k60, ddsim.statevector: holds the phase of its small-angle gate: the share of the step from the state without that gate to the ideal (1 holds it, 0 drops it) — measured -4.3e-78. drops it: the angle, 5.45e-18 rad, is below its parameter tolerance of 1e-13 rad, where it may read an angle as 0

Versions: cirq-core 1.7.0, mqt.core 3.11.0, mqt.ddsim 2.7.0, numpy 2.5.4, python 3.12.16, qiskit 2.6.2, qiskit-aer 0.18.2, qsimcirq 0.22.1, quimb 1.15.0, scipy 1.18.1, stim 1.16.0.

performance-linux-x86_64.json

Host: AMD RYZEN AI MAX+ 395 w/ Radeon 8060S, Ubuntu 26.04.1 LTS, 32 logical CPUs, 123 GiB, x86_64. Each figure is the engine's own call, the median of 5 runs (the minimum in brackets). Memory is the peak resident set of the process that ran the engine, and the baseline the same given a two-qubit job. Ops are the operations the engine ran, where it reports them.

• aer.* seconds: the simulator's own time for the experiment, as its result states it (time_taken); the conversion of the circuit to its own form is outside it.
• ddsim.* seconds: the run call, which converts the circuit to its own form inside it.
• qsim.* seconds: the simulate call, which converts the circuit to its own form inside it.
• quimb.* seconds: the building of the circuit object, which applies each gate as it is added, and the read-out.
• stim.* seconds: the tableau's or sampler's run, after the circuit text is parsed.
• wai.* seconds: the engine's run, after the program is read and lowered (the driver's own clock).

case	engine	threads	ops	seconds, median (min)	peak MiB (baseline)	load average before → after
qft_n20	wai.statevector	all (32)	230	0.0363 (0.0347)	26 (5)	13.9 → 13.9
qft_n20	aer.statevector	32	231	0.819 (0.737)	116 (100)	13.9 → 13.7
qft_n20	qsim.statevector	32		4.78 (3.16)	196 (188)	14.6 → 28.9
qft_n20	ddsim.statevector	not set		0.349 (0.329)	132 (99)	28.0 → 28.0
qft_n20	wai.exact	1 (no threads)	250	0.27 (0.26)	38 (5)	28.0 → 27.2
qft_n20	quimb.dense	not set		4.92 (3.99)	291 (195)	27.2 → 31.9
qft_n20	wai.statevector	1 (pinned)	230	0.372 (0.352)	23 (5)	31.9 → 31.2
qft_n20	aer.statevector	1 (pinned)	231	0.445 (0.294)	116 (100)	31.2 → 31.4
qft_n20	qsim.statevector	1 (pinned)		0.0454 (0.0434)	195 (188)	31.4 → 29.3
qft_n20	ddsim.statevector	1 (pinned)		0.562 (0.5)	132 (99)	29.3 → 27.7
qft_n20	wai.exact	1 (pinned)	250	0.393 (0.349)	38 (5)	27.7 → 27.7
qft_n20	quimb.dense	1 (pinned)		1.37 (1.26)	276 (195)	27.7 → 25.0
qft_n24	wai.statevector	all (32)	324	0.45 (0.441)	279 (5)	25.0 → 26.2
qft_n24	aer.statevector	32	325	3.9 (1.6)	356 (100)	26.2 → 29.5
qft_n24	qsim.statevector	32		2.56 (1.91)	315 (188)	29.5 → 30.9
qft_n24	ddsim.statevector	not set		6.22 (5.34)	612 (99)	30.9 → 25.7
qft_n24	wai.exact	1 (no threads)	348	6.6 (6.48)	518 (5)	25.7 → 18.8
qft_n24	wai.statevector	1 (pinned)	324	5.82 (5.35)	263 (5)	18.8 → 20.5
qft_n24	aer.statevector	1 (pinned)	325	8.17 (6.6)	356 (100)	20.5 → 21.2
qft_n24	qsim.statevector	1 (pinned)		1.9 (1.63)	315 (188)	21.2 → 19.6
qft_n24	ddsim.statevector	1 (pinned)		7.36 (6.23)	612 (99)	19.6 → 18.5
qft_n24	wai.exact	1 (pinned)	348	7.8 (7)	518 (5)	18.5 → 14.9
qft_n26	wai.statevector	all (32)	377	2.14 (2.05)	1047 (5)	14.9 → 19.9
qft_n26	aer.statevector	32	378	3.97 (3.7)	1124 (100)	19.9 → 23.6
qft_n26	qsim.statevector	32		5.62 (3.93)	700 (188)	23.6 → 31.5
qft_n28	wai.statevector	all (32)	434	14.2 (12.4)	4118 (5)	31.5 → 44.7
qft_n28	aer.statevector	32	435	16.8 (15.3)	4198 (100)	44.7 → 41.8
qft_n28	qsim.statevector	32		12.7 (11.9)	2234 (188)	41.8 → 38.5
rotations_n20	wai.statevector	all (32)	590	0.0963 (0.0761)	27 (5)	38.5 → 38.5
rotations_n20	aer.statevector	32	591	1.36 (1.06)	120 (100)	38.5 → 40.0
rotations_n20	qsim.statevector	32		1.42 (1.22)	195 (188)	40.0 → 38.4
rotations_n20	quimb.dense	not set		15.7 (9.5)	290 (195)	38.4 → 54.5
rotations_n20	wai.statevector	1 (pinned)	590	0.613 (0.481)	23 (5)	54.5 → 52.2
rotations_n20	aer.statevector	1 (pinned)	591	0.876 (0.709)	120 (100)	52.2 → 50.1
rotations_n20	qsim.statevector	1 (pinned)		0.0458 (0.0437)	196 (188)	50.1 → 45.3
rotations_n20	quimb.dense	1 (pinned)		2.25 (2.2)	276 (195)	45.3 → 38.4
rotations_n24	wai.statevector	all (32)	710	0.892 (0.869)	282 (5)	38.4 → 37.3
rotations_n24	aer.statevector	32	711	2.65 (2.44)	356 (100)	37.3 → 38.9
rotations_n24	qsim.statevector	32		1.71 (1.52)	315 (188)	38.9 → 38.1
rotations_n24	wai.statevector	1 (pinned)	710	10.4 (9.72)	263 (5)	38.1 → 21.1
rotations_n24	aer.statevector	1 (pinned)	711	15.1 (13.3)	356 (100)	21.1 → 13.5
rotations_n24	qsim.statevector	1 (pinned)		1.54 (1.52)	316 (188)	13.5 → 13.6
rotations_n26	wai.statevector	all (32)	770	4.12 (3.94)	1050 (5)	13.6 → 23.3
rotations_n26	aer.statevector	32	771	8.7 (8)	1123 (100)	23.3 → 33.4
rotations_n26	qsim.statevector	32		4.15 (4.07)	700 (188)	33.4 → 35.5
grover_n10	wai.statevector	all (32)	114	0.000998 (0.000871)	6 (5)	35.5 → 35.5
grover_n10	wai.exact	1 (no threads)	26686	0.0363 (0.0331)	17 (5)	35.5 → 35.5
grover_n10	wai.exact_whole	1 (no threads)	114	0.000195 (0.000162)	6 (5)	35.5 → 35.5
grover_n10	aer.statevector	32	123	0.000298 (0.000293)	100 (100)	35.5 → 35.5
grover_n10	qsim.statevector	32		0.0014 (0.00129)	188 (188)	35.5 → 33.5
grover_n10	ddsim.statevector	not set		0.00883 (0.00609)	100 (99)	33.5 → 33.5
grover_n10	wai.statevector	1 (pinned)	114	0.000446 (0.000321)	6 (5)	33.5 → 33.5
grover_n10	wai.exact	1 (pinned)	26686	0.0507 (0.0317)	17 (5)	33.5 → 33.5
grover_n10	wai.exact_whole	1 (pinned)	114	0.000273 (0.000271)	6 (5)	33.5 → 31.6
grover_n10	aer.statevector	1 (pinned)	123	0.000298 (0.000289)	100 (100)	31.6 → 31.6
grover_n10	qsim.statevector	1 (pinned)		0.00137 (0.00128)	187 (188)	31.6 → 29.7
grover_n10	ddsim.statevector	1 (pinned)		0.00742 (0.0058)	100 (99)	29.7 → 27.8
grover_n14	wai.statevector	all (32)	158	0.00241 (0.00227)	6 (5)	27.8 → 27.8
grover_n14	wai.exact	1 (no threads)	557146	11.4 (10.6)	242 (5)	27.8 → 14.9
grover_n14	wai.exact_whole	1 (no threads)	158	0.00354 (0.00352)	6 (5)	14.9 → 14.9
grover_n14	aer.statevector	32	167	0.00298 (0.00286)	100 (100)	14.9 → 14.9
grover_n14	qsim.statevector	32		0.0773 (0.0661)	188 (188)	14.9 → 14.7
grover_n14	ddsim.statevector	not set		0.00672 (0.0066)	99 (99)	14.7 → 14.7
grover_n14	wai.statevector	1 (pinned)	158	0.00225 (0.00156)	6 (5)	14.7 → 14.7
grover_n14	wai.exact	1 (pinned)	557146	17.4 (13.8)	242 (5)	14.7 → 12.9
grover_n14	wai.exact_whole	1 (pinned)	158	0.00343 (0.0034)	6 (5)	12.9 → 12.9
grover_n14	aer.statevector	1 (pinned)	167	0.00296 (0.00281)	100 (100)	12.9 → 12.9
grover_n14	qsim.statevector	1 (pinned)		0.0018 (0.0018)	187 (188)	12.9 → 12.7
grover_n14	ddsim.statevector	1 (pinned)		0.0073 (0.00706)	100 (99)	12.7 → 12.7
grover_n16	wai.statevector	all (32)	180	0.00857 (0.0083)	7 (5)	12.7 → 12.7
grover_n16	wai.exact_whole	1 (no threads)	180	0.0169 (0.0162)	7 (5)	12.7 → 12.7
grover_n16	aer.statevector	32	189	0.462 (0.437)	101 (100)	12.7 → 15.2
grover_n16	qsim.statevector	32		0.328 (0.288)	189 (188)	15.2 → 15.1
grover_n16	ddsim.statevector	not set		0.00875 (0.0085)	101 (99)	15.1 → 15.1
grover_n16	wai.statevector	1 (pinned)	180	0.00965 (0.0096)	7 (5)	15.1 → 15.1
grover_n16	wai.exact_whole	1 (pinned)	180	0.0281 (0.0269)	8 (5)	15.1 → 15.1
grover_n16	aer.statevector	1 (pinned)	189	0.0186 (0.0182)	101 (100)	15.1 → 14.6
grover_n16	qsim.statevector	1 (pinned)		0.00342 (0.00234)	188 (188)	14.6 → 14.4
grover_n16	ddsim.statevector	1 (pinned)		0.0115 (0.0113)	102 (99)	14.4 → 14.2
grover_n20	wai.statevector	all (32)	224	0.0268 (0.0259)	25 (5)	14.2 → 14.2
grover_n20	wai.exact_whole	1 (no threads)	224	0.34 (0.339)	38 (5)	14.2 → 14.2
grover_n20	aer.statevector	32	233	0.72 (0.61)	124 (100)	14.2 → 16.7
grover_n20	qsim.statevector	32		0.429 (0.316)	195 (188)	16.7 → 16.6
grover_n20	ddsim.statevector	not set		0.0332 (0.0325)	132 (99)	16.6 → 16.5
grover_n20	wai.statevector	1 (pinned)	224	0.204 (0.204)	22 (5)	16.5 → 16.5
grover_n20	wai.exact_whole	1 (pinned)	224	0.586 (0.583)	38 (5)	16.5 → 16.5
grover_n20	aer.statevector	1 (pinned)	233	0.35 (0.35)	124 (100)	16.5 → 16.2
grover_n20	qsim.statevector	1 (pinned)		0.0161 (0.0133)	195 (188)	16.2 → 15.5
grover_n20	ddsim.statevector	1 (pinned)		0.0423 (0.032)	132 (99)	15.5 → 15.5
clifford_n1000	wai.stabilizer	1 (no threads)	35082	0.000968 (0.000961)	90 (5)	15.5 → 15.6
clifford_n1000	stim.tableau	not set		0.002 (0.00126)	40 (24)	15.6 → 15.6
clifford_n1000	aer.stabilizer	32	35083	0.0184 (0.0161)	207 (121)	15.6 → 15.7
clifford_n5000	wai.stabilizer	1 (no threads)	58265	0.00666 (0.00655)	142 (5)	15.9 → 15.6
clifford_n5000	stim.tableau	not set		0.00839 (0.00829)	65 (24)	15.6 → 15.6
surface_z_d5_sample	wai.qec	1 (no threads)		0.0762 (0.0737)	19 (4)	15.6 → 15.7
surface_z_d5_sample	stim.tableau	not set		0.348 (0.337)	160 (38)	15.7 → 15.7
surface_z_d5_sample	wai.qec	1 (pinned)		0.0949 (0.0812)	19 (4)	15.7 → 15.7
surface_z_d5_sample	stim.tableau	1 (pinned)		0.292 (0.273)	160 (38)	15.7 → 15.5
surface_z_d5_model	wai.qec	1 (no threads)		0.00185 (0.00182)	7 (5)	15.5 → 15.5
surface_z_d5_model	stim.tableau	not set		0.00321 (0.00319)	26 (25)	15.5 → 15.5
surface_z_d11_sample	wai.qec	1 (no threads)		0.817 (0.811)	163 (4)	15.5 → 15.5
surface_z_d11_sample	stim.tableau	not set		3.19 (3.05)	853 (38)	15.5 → 16.8
surface_z_d11_sample	wai.qec	1 (pinned)		1.06 (1.06)	163 (4)	16.8 → 16.8
surface_z_d11_sample	stim.tableau	1 (pinned)		4.05 (3.33)	853 (38)	16.8 → 16.8
surface_z_d11_model	wai.qec	1 (no threads)		0.0442 (0.0373)	26 (5)	16.8 → 16.8
surface_z_d11_model	stim.tableau	not set		0.0542 (0.0509)	40 (25)	16.8 → 16.8
qft_n40_amplitudes	wai.mps	1 (no threads)		0.0114 (0.0107)	11 (7)	16.8 → 16.8
qft_n40_amplitudes	aer.mps	32	861	0.00371 (0.00352)	105 (99)	16.8 → 16.6
qft_n40_amplitudes	quimb.mps	not set		1.98 (1.91)	206 (200)	16.6 → 18.2
qft_n40_amplitudes	wai.tensor_network	all (32)		0.155 (0.15)	12 (7)	18.2 → 17.9
qft_n40_amplitudes	quimb.tensor_network	not set		1.08 (0.961)	220 (205)	17.9 → 17.8

performance-macos-arm64.json

Host: Apple M3 Ultra, macOS 27.2, 32 logical CPUs, 256 GiB, arm64. Each figure is the engine's own call, the median of 5 runs (the minimum in brackets). Memory is the peak resident set of the process that ran the engine, and the baseline the same given a two-qubit job. Ops are the operations the engine ran, where it reports them.

• aer.* seconds: the simulator's own time for the experiment, as its result states it (time_taken); the conversion of the circuit to its own form is outside it.
• ddsim.* seconds: the run call, which converts the circuit to its own form inside it.
• qsim.* seconds: the simulate call, which converts the circuit to its own form inside it.
• quimb.* seconds: the building of the circuit object, which applies each gate as it is added, and the read-out.
• stim.* seconds: the tableau's or sampler's run, after the circuit text is parsed.
• wai.* seconds: the engine's run, after the program is read and lowered (the driver's own clock).

case	engine	threads	ops	seconds, median (min)	peak MiB (baseline)	load average before → after
qft_n20	wai.statevector	all (32)	230	0.0326 (0.0307)	28 (4)	84.8 → 84.9
qft_n20	aer.statevector	32	231	0.0841 (0.0824)	101 (83)	84.9 → 84.9
qft_n20	qsim.statevector	32		0.0905 (0.0771)	197 (188)	84.9 → 90.7
qft_n20	ddsim.statevector	not set		0.145 (0.142)	119 (86)	89.8 → 89.8
qft_n20	wai.exact	1 (no threads)	250	0.242 (0.231)	37 (3)	89.8 → 89.8
qft_n20	quimb.dense	not set		1.6 (1.49)	256 (158)	89.8 → 87.8
qft_n24	wai.statevector	all (32)	324	0.585 (0.54)	308 (4)	87.8 → 88.6
qft_n24	aer.statevector	32	325	1.02 (0.944)	341 (83)	88.6 → 88.7
qft_n24	qsim.statevector	32		0.825 (0.777)	317 (188)	88.7 → 89.6
qft_n24	ddsim.statevector	not set		2.24 (2.2)	599 (86)	89.6 → 87.0
qft_n24	wai.exact	1 (no threads)	348	5.76 (5.49)	518 (3)	87.0 → 91.1
qft_n26	wai.statevector	all (32)	377	2.66 (2.16)	1070 (4)	91.1 → 107.2
qft_n26	aer.statevector	32	378	2.28 (2.08)	1109 (83)	107.2 → 108.9
qft_n26	qsim.statevector	32		2.77 (2.72)	701 (188)	108.9 → 108.4
qft_n28	wai.statevector	all (32)	434	10.2 (10.2)	4143 (4)	108.4 → 129.7
qft_n28	aer.statevector	32	435	7.04 (7.02)	4181 (83)	129.7 → 133.5
qft_n28	qsim.statevector	32		9.45 (9.41)	2238 (188)	133.5 → 107.2
rotations_n20	wai.statevector	all (32)	590	0.0699 (0.0663)	29 (4)	107.2 → 107.2
rotations_n20	aer.statevector	32	591	0.195 (0.166)	102 (83)	107.2 → 107.1
rotations_n20	qsim.statevector	32		0.143 (0.0835)	199 (188)	107.1 → 100.2
rotations_n20	quimb.dense	not set		2.84 (2.82)	236 (158)	100.2 → 99.3
rotations_n24	wai.statevector	all (32)	710	0.925 (0.901)	303 (4)	99.3 → 102.6
rotations_n24	aer.statevector	32	711	1.58 (1.51)	342 (83)	102.6 → 102.4
rotations_n24	qsim.statevector	32		0.601 (0.585)	318 (188)	102.4 → 100.1
rotations_n26	wai.statevector	all (32)	770	2.36 (2.3)	1074 (4)	100.1 → 98.1
rotations_n26	aer.statevector	32	771	3.89 (3.83)	1110 (83)	98.1 → 94.5
rotations_n26	qsim.statevector	32		1.99 (1.98)	702 (188)	94.5 → 91.2
grover_n10	wai.statevector	all (32)	114	0.000183 (0.000175)	4 (4)	91.2 → 91.2
grover_n10	wai.exact	1 (no threads)	26686	0.0264 (0.0264)	17 (3)	91.2 → 91.2
grover_n10	wai.exact_whole	1 (no threads)	114	0.00015 (0.00015)	4 (3)	91.2 → 91.2
grover_n10	aer.statevector	32	123	0.000252 (0.000226)	84 (83)	91.2 → 89.9
grover_n10	qsim.statevector	32		0.00337 (0.00307)	189 (188)	89.9 → 88.5
grover_n10	ddsim.statevector	not set		0.00703 (0.00571)	87 (86)	88.5 → 88.5
grover_n14	wai.statevector	all (32)	158	0.00223 (0.00221)	4 (4)	88.5 → 88.5
grover_n14	wai.exact	1 (no threads)	557146	9.18 (9.1)	261 (3)	88.5 → 73.9
grover_n14	wai.exact_whole	1 (no threads)	158	0.00322 (0.0032)	4 (3)	73.9 → 73.9
grover_n14	aer.statevector	32	167	0.00316 (0.00313)	84 (83)	73.9 → 75.7
grover_n14	qsim.statevector	32		0.0138 (0.0123)	189 (188)	75.7 → 71.6
grover_n14	ddsim.statevector	not set		0.00643 (0.00622)	88 (86)	71.6 → 71.6
grover_n16	wai.statevector	all (32)	180	0.011 (0.0106)	7 (4)	71.6 → 71.6
grover_n16	wai.exact_whole	1 (no threads)	180	0.0148 (0.0146)	6 (3)	71.6 → 71.6
grover_n16	aer.statevector	32	189	0.0194 (0.016)	86 (83)	71.6 → 73.7
grover_n16	qsim.statevector	32		0.0151 (0.0134)	189 (188)	73.7 → 73.4
grover_n16	ddsim.statevector	not set		0.00702 (0.00667)	89 (86)	73.4 → 73.4
grover_n20	wai.statevector	all (32)	224	0.0301 (0.0286)	28 (4)	73.4 → 79.4
grover_n20	wai.exact_whole	1 (no threads)	224	0.322 (0.319)	36 (3)	79.4 → 79.4
grover_n20	aer.statevector	32	233	0.0588 (0.0546)	108 (83)	79.4 → 79.4
grover_n20	qsim.statevector	32		0.0254 (0.0247)	197 (188)	79.4 → 74.8
grover_n20	ddsim.statevector	not set		0.0159 (0.0157)	119 (86)	74.8 → 74.8
clifford_n1000	wai.stabilizer	1 (no threads)	35082	0.00106 (0.00103)	95 (3)	74.8 → 74.8
clifford_n1000	stim.tableau	not set		0.0017 (0.00163)	46 (27)	74.8 → 74.8
clifford_n1000	aer.stabilizer	32	35083	0.00963 (0.00854)	193 (99)	74.8 → 69.7
clifford_n5000	wai.stabilizer	1 (no threads)	58265	0.0127 (0.0123)	168 (3)	77.3 → 77.3
clifford_n5000	stim.tableau	not set		0.0441 (0.0331)	74 (27)	77.3 → 77.3
surface_z_d5_sample	wai.qec	1 (no threads)		0.0897 (0.0872)	18 (3)	77.3 → 76.9
surface_z_d5_sample	stim.tableau	not set		0.224 (0.219)	123 (38)	76.9 → 76.9
surface_z_d5_model	wai.qec	1 (no threads)		0.00231 (0.00226)	6 (3)	76.9 → 76.9
surface_z_d5_model	stim.tableau	not set		0.00279 (0.00269)	28 (28)	76.9 → 76.9
surface_z_d11_sample	wai.qec	1 (no threads)		0.997 (0.981)	163 (3)	76.9 → 77.1
surface_z_d11_sample	stim.tableau	not set		2.18 (2.18)	938 (38)	77.1 → 75.9
surface_z_d11_model	wai.qec	1 (no threads)		0.0264 (0.0243)	33 (3)	75.9 → 74.7
surface_z_d11_model	stim.tableau	not set		0.04 (0.0399)	47 (28)	74.7 → 74.7
qft_n40_amplitudes	wai.mps	1 (no threads)		0.0167 (0.0159)	10 (5)	74.7 → 74.7
qft_n40_amplitudes	aer.mps	32	861	0.00552 (0.00523)	86 (83)	74.7 → 74.7
qft_n40_amplitudes	quimb.mps	not set		1.93 (1.92)	167 (160)	74.7 → 64.9
qft_n40_amplitudes	wai.tensor_network	all (32)		0.197 (0.194)	12 (5)	65.7 → 65.7
qft_n40_amplitudes	quimb.tensor_network	not set		0.934 (0.916)	180 (167)	65.7 → 70.7

before-after-macos-arm64.json

Host: Apple M3 Ultra, macOS 27.2, 32 logical CPUs, 256 GiB, arm64. The same harness and circuits, with the engine under test built before and after a gate under many controls stood whole on its state vector, one build straight after the other: each figure is the engine's own call, the median of 5 runs (the minimum in brackets), and ops are the operations it ran.

wai.exact_whole's before rows ran a build with no such option, so they are standard gates, as their ops show. The last column is the one-minute load as each build's row began.

case	engine	threads	before: ops	before: seconds, median (min)	after: ops	after: seconds, median (min)	load average, before build → after build
qft_n20	wai.statevector	all (32)	230	0.0227 (0.0223)	230	0.0326 (0.0307)	30.9 → 84.8
qft_n20	aer.statevector	32	231	0.0708 (0.0672)	231	0.0841 (0.0824)	30.9 → 84.9
qft_n20	qsim.statevector	32		0.0564 (0.0557)		0.0905 (0.0771)	30.9 → 84.9
qft_n20	ddsim.statevector	not set		0.14 (0.14)		0.145 (0.142)	29.4 → 89.8
qft_n20	wai.exact	1 (no threads)	250	0.22 (0.214)	250	0.242 (0.231)	32.9 → 89.8
qft_n20	quimb.dense	not set		1.41 (1.38)		1.6 (1.49)	32.9 → 89.8
qft_n24	wai.statevector	all (32)	324	0.474 (0.468)	324	0.585 (0.54)	32.5 → 87.8
qft_n24	aer.statevector	32	325	0.565 (0.532)	325	1.02 (0.944)	32.5 → 88.6
qft_n24	qsim.statevector	32		0.555 (0.528)		0.825 (0.777)	33.5 → 88.7
qft_n24	ddsim.statevector	not set		2.12 (2.12)		2.24 (2.2)	39.0 → 89.6
qft_n24	wai.exact	1 (no threads)	348	5.01 (5)	348	5.76 (5.49)	38.2 → 87.0
qft_n26	wai.statevector	all (32)	377	2.1 (2.06)	377	2.66 (2.16)	42.0 → 91.1
qft_n26	aer.statevector	32	378	1.84 (1.76)	378	2.28 (2.08)	48.6 → 107.2
qft_n26	qsim.statevector	32		2.11 (2.09)		2.77 (2.72)	51.3 → 108.9
qft_n28	wai.statevector	all (32)	434	9.44 (9.32)	434	10.2 (10.2)	44.6 → 108.4
qft_n28	aer.statevector	32	435	6.72 (6.58)	435	7.04 (7.02)	57.7 → 129.7
qft_n28	qsim.statevector	32		8.98 (8.9)		9.45 (9.41)	63.9 → 133.5
rotations_n20	wai.statevector	all (32)	590	0.0619 (0.0609)	590	0.0699 (0.0663)	55.2 → 107.2
rotations_n20	aer.statevector	32	591	0.13 (0.123)	591	0.195 (0.166)	55.2 → 107.2
rotations_n20	qsim.statevector	32		0.0568 (0.0534)		0.143 (0.0835)	55.2 → 107.1
rotations_n20	quimb.dense	not set		2.61 (2.6)		2.84 (2.82)	53.9 → 100.2
rotations_n24	wai.statevector	all (32)	710	0.982 (0.909)	710	0.925 (0.901)	53.7 → 99.3
rotations_n24	aer.statevector	32	711	1.3 (1.17)	711	1.58 (1.51)	59.2 → 102.6
rotations_n24	qsim.statevector	32		0.526 (0.463)		0.601 (0.585)	59.4 → 102.4
rotations_n26	wai.statevector	all (32)	770	2.32 (2.3)	770	2.36 (2.3)	53.0 → 100.1
rotations_n26	aer.statevector	32	771	3.92 (3.63)	771	3.89 (3.83)	55.1 → 98.1
rotations_n26	qsim.statevector	32		1.77 (1.66)		1.99 (1.98)	57.8 → 94.5
grover_n10	wai.statevector	all (32)	8290	0.0156 (0.0152)	114	0.000183 (0.000175)	49.9 → 91.2
grover_n10	wai.exact	1 (no threads)	26686	0.0251 (0.0247)	26686	0.0264 (0.0264)	49.9 → 91.2
grover_n10	wai.exact_whole	1 (no threads)	26686	0.0249 (0.0248)	114	0.00015 (0.00015)	49.9 → 91.2
grover_n10	aer.statevector	32	123	0.000224 (0.000219)	123	0.000252 (0.000226)	47.0 → 91.2
grover_n10	qsim.statevector	32		0.00315 (0.00293)		0.00337 (0.00307)	47.0 → 89.9
grover_n10	ddsim.statevector	not set		0.0054 (0.00457)		0.00703 (0.00571)	44.2 → 88.5
grover_n14	wai.statevector	all (32)	131214	3 (2.97)	158	0.00223 (0.00221)	44.2 → 88.5
grover_n14	wai.exact	1 (no threads)	557146	8.48 (8.39)	557146	9.18 (9.1)	46.0 → 88.5
grover_n14	wai.exact_whole	1 (no threads)	557146	8.45 (8.37)	158	0.00322 (0.0032)	35.3 → 73.9
grover_n14	aer.statevector	32	167	0.00318 (0.00303)	167	0.00316 (0.00313)	35.0 → 73.9
grover_n14	qsim.statevector	32		0.0114 (0.0102)		0.0138 (0.0123)	34.7 → 75.7
grover_n14	ddsim.statevector	not set		0.0059 (0.00558)		0.00643 (0.00622)	34.6 → 71.6
grover_n16	wai.statevector	all (32)	524452	16.3 (15.5)	180	0.011 (0.0106)	34.6 → 71.6
grover_n16	wai.exact_whole	1 (no threads)	2490472	151 (151)	180	0.0148 (0.0146)	35.1 → 71.6
grover_n16	aer.statevector	32	189	0.0168 (0.0137)	189	0.0194 (0.016)	66.7 → 71.6
grover_n16	qsim.statevector	32		0.0174 (0.0158)		0.0151 (0.0134)	66.7 → 73.7
grover_n16	ddsim.statevector	not set		0.00663 (0.00625)		0.00702 (0.00667)	70.6 → 73.4
grover_n20	wai.statevector	all (32)		not run: line 34: modifiers add at most 16 controls here [Q208]	224	0.0301 (0.0286)	70.6 → 73.4
grover_n20	wai.exact_whole	1 (no threads)		not run: line 34: modifiers add at most 16 controls here [Q208]	224	0.322 (0.319)	70.6 → 79.4
grover_n20	aer.statevector	32	233	0.063 (0.0566)	233	0.0588 (0.0546)	70.6 → 79.4
grover_n20	qsim.statevector	32		0.032 (0.0248)		0.0254 (0.0247)	70.6 → 79.4
grover_n20	ddsim.statevector	not set		0.0179 (0.0166)		0.0159 (0.0157)	74.4 → 74.8
clifford_n1000	wai.stabilizer	1 (no threads)	35082	0.00102 (0.00101)	35082	0.00106 (0.00103)	78.1 → 74.8
clifford_n1000	stim.tableau	not set		0.00163 (0.0016)		0.0017 (0.00163)	78.1 → 74.8
clifford_n1000	aer.stabilizer	32	35083	0.0088 (0.00803)	35083	0.00963 (0.00854)	78.1 → 74.8
clifford_n5000	wai.stabilizer	1 (no threads)	58265	0.0124 (0.0112)	58265	0.0127 (0.0123)	85.1 → 77.3
clifford_n5000	stim.tableau	not set		0.0389 (0.0299)		0.0441 (0.0331)	85.6 → 77.3
surface_z_d5_sample	wai.qec	1 (no threads)		0.0913 (0.0879)		0.0897 (0.0872)	85.6 → 77.3
surface_z_d5_sample	stim.tableau	not set		0.228 (0.222)		0.224 (0.219)	85.6 → 76.9
surface_z_d5_model	wai.qec	1 (no threads)		0.00231 (0.00223)		0.00231 (0.00226)	86.8 → 76.9
surface_z_d5_model	stim.tableau	not set		0.00329 (0.00271)		0.00279 (0.00269)	86.8 → 76.9
surface_z_d11_sample	wai.qec	1 (no threads)		0.968 (0.962)		0.997 (0.981)	86.8 → 76.9
surface_z_d11_sample	stim.tableau	not set		2.21 (2.19)		2.18 (2.18)	88.7 → 77.1
surface_z_d11_model	wai.qec	1 (no threads)		0.0251 (0.0245)		0.0264 (0.0243)	87.2 → 75.9
surface_z_d11_model	stim.tableau	not set		0.0443 (0.0405)		0.04 (0.0399)	87.2 → 74.7
qft_n40_amplitudes	wai.mps	1 (no threads)		0.0163 (0.0161)		0.0167 (0.0159)	89.1 → 74.7
qft_n40_amplitudes	aer.mps	32	861	0.00552 (0.00538)	861	0.00552 (0.00523)	89.1 → 74.7
qft_n40_amplitudes	quimb.mps	not set		1.96 (1.93)		1.93 (1.92)	89.1 → 74.7
qft_n40_amplitudes	wai.tensor_network	all (32)		0.188 (0.187)		0.197 (0.194)	86.7 → 65.7
qft_n40_amplitudes	quimb.tensor_network	not set		0.985 (0.972)		0.934 (0.916)	85.4 → 65.7

Limits

• The bounds are loose. They are worst cases, so an error smaller than a bound passes, and a double-precision engine usually measures a hundred times or more below its bound. The ratios in the tables say how much room each check left. The sharp checks are the exact ones: the referee's bit-for-bit rerun, the bit-exact op-log against the circuit, the held-or-dropped check of each small-angle gate, stabilizer groups as bytes, outcomes of probability 0, and symptom sets.
• The integer state reference stops at 14 qubits. Wider circuits are held to closed forms (QFT, GHZ), to mirror circuits, and to each other.
• Sampling is checked by marginals and by L1 distance. Each outcome's count is held to its probability, and the whole distribution to the exact one; the independence of shots is not tested. At the family-wise rate of 10^-6 a test fails on a deviation of about six standard deviations: on 100,000 shots of a fair outcome, about 1,000 shots, and in L1 distance about 0.02, or 0.045 over 257 outcomes at 200,000 shots. A smaller fault passes; each check records its bound. At 40 qubits a sampler confined to a smaller space inside the right one, with every bit still free, is not seen.
• Not compared at all: the engine's Pauli propagation, cut-rank, stabilizer-rank, trajectory, pulse and mitigation methods; its GPU and other fabric members; single-precision GPU state vectors of any engine; decoders; noise beyond depolarizing channels in closed form.
• What the engine under test refuses is listed in corpus/expected-refusals.json, with the words it must give. Its frame simulator refuses a Pauli channel whose exclusive outcomes no independent flips can give (PAULI_CHANNEL_1(0.1, 0.1, 0), whose Z would need a rate below zero), with its line: a detector error model of independent mechanisms cannot state it either. A general Pauli channel it can take (qec_pauli_channel) it writes as the independent flips with the channel's law, and its model matches the other engine's mechanism by mechanism. A job naming method mps runs under a meter of the bonds it holds (mps_qft_n30_bond256) and stops with [J011] if those pass one job's work; one naming auto keeps to the estimate at the bond asked for.
• What the adapters do not drive. The qsim adapter runs no measurement in the middle of a circuit, so those entries are not run for it; that is this adapter's limit. aer.mps is not run at 100 qubits: its target holds 63. An engine that has not answered a circuit within 900 s is stopped, with every process it started, and is recorded as not run for it, or as a failure when it is the engine under test.
• Platforms. The pinned wheels exist together for macOS on arm64 and Linux on x86_64 only.
• Performance is one shared host a file, on one day. Other work on the host moves a timing by several times; the load average before and after each case is beside it. Timers do not all cover the same work: the list above each table says what each covers.
• One author wrote the adapters, the references and the tolerances. A reproduction by someone else, and adapters reviewed by each engine's own users, are what this page still needs.

A provider of the open protocol

provider/seeded_statevector_provider.py wraps a third-party state-vector simulator as a process provider of wai.quantum.provider/1 (the extension). It is also the template for a device of your own: replace the Device class's run with a call to your device.

wai-quantum providers add sim --kind process -- python3 -I provider/seeded_statevector_provider.py
wai-quantum providers test sim        # the protocol's conformance vectors
wai-quantum run bell.qasm --provider sim --shots 1000 --json > job.json
wai-quantum verify job.json

provider/test.sh does exactly that in a directory of its own, as part of reproduce.sh and of CI: every conformance vector passes, the counts are sealed as measured on a simulator, the receipt verifies, and the same job gives the same counts twice (the device is seeded; on the two hosts here it gave the same counts, too).

CI

.github/workflows/wai-quantum-cli.yml runs the CI tier on Linux (the circuits marked ci in the manifest: every kind of check but the pairwise comparison of wide circuits), the self-test, the provider template, and tools/report.py --check, which holds this page's results section to results/. The full tier is the command above, run by hand.

Files

path	what
reproduce.sh	the one command
env/requirements.in, env/requirements.lock	the pinned engines, and every wheel's hash for macOS and Linux
corpus/	the manifest, the committed programs and error-correction circuits, the hand-written ones, the refusals the engine under test may give, and the known disagreements with their reproducers
tools/corpus.py	the circuits
tools/exact.py	the references
tools/tolerance.py	every tolerance and where it comes from
tools/engines.py	one adapter per engine and method
tools/compare.py	the run: references, tolerances, injected faults, the results file
tools/perf.py	run time and peak memory
tools/report.py	this page's results section
tools/before_after.py	one host's timings of two builds side by side, from tools/perf.py run on each
provider/	the provider template and its test
results/	the committed runs