00cfcf0530
`wai-quantum eval run` asks a model each task of a set, once and at temperature 0. The model is asked either for the program alone or for an answer in the answer view layer. The engine then scores what it writes: whether the answer reads, the verdict's level against the task's intent, whether the task is solved, and the joules per solved task where the model server's process energy can be read. Repair rounds send the engine's codes back for a corrected answer, and the report keeps the first try's counts. A model server that is neither on this machine nor on a private network is refused unless allowed. The report ranks nothing, and no report is committed. The task set (wai/quantum-conformance/eval/tasks.json) is fourteen tasks from the intent conformance set and the worked examples. Each has a reference program that a test holds to its intent by proof. `eval corpus` writes each task's answer four ways: the program alone, the layer whole, the layer with the request alone, and a job's JSON. Noise that two calls share is named once. token_cost.py counts each form with a model's own vocabulary. Its report on three open-weight vocabularies is committed beside the corpus. The smoke checks run a scripted model through the harness, and CI names each harness test. Co-Authored-By: Claude Opus 5.5 <[email protected]>
439 lines
14 KiB
JSON
439 lines
14 KiB
JSON
{
|
|
"format": "wai.quantum-intent-conformance-bundle/1",
|
|
"about": "The model harness's task set (wai-quantum eval run --tasks): each case what a model is asked (ask), the intent its program is held to, the run, and a reference program (qasm) that a test holds to the intent by proof. Built from the intent conformance set's cases (wai/quantum-conformance/intents/) and the worked examples (wai_quantum::engine::guide::examples), each named in from, with a few more of the first states of a circuits course.",
|
|
"cases": [
|
|
{
|
|
"name": "plus",
|
|
"from": [
|
|
"new: the first state of the circuits curriculum"
|
|
],
|
|
"covers": [
|
|
"Put one qubit in an equal superposition of 0 and 1, so that half the shots read 0 and half read 1."
|
|
],
|
|
"ask": "Put one qubit in an equal superposition of 0 and 1, so that half the shots read 0 and half read 1. Use 1 qubit. Measure every qubit at the end into a classical bit of the same index (c = measure q;); outcomes are written with the highest bit first.",
|
|
"qubits": 1,
|
|
"qasm": "OPENQASM 3.0;\nqubit[1] q;\nbit[1] c;\nh q[0];\nc = measure q;\n",
|
|
"run": {
|
|
"shots": 100
|
|
},
|
|
"intent": [
|
|
{
|
|
"distribution": {
|
|
"0": 0.5,
|
|
"1": 0.5
|
|
}
|
|
}
|
|
],
|
|
"expect": {
|
|
"holds": true,
|
|
"levels": [
|
|
"proof",
|
|
"evidence"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"name": "bell",
|
|
"from": [
|
|
"intents: bell-distribution",
|
|
"examples: bell"
|
|
],
|
|
"covers": [
|
|
"Prepare a Bell pair on two qubits: half the shots read 00 and half 11, never 01 or 10."
|
|
],
|
|
"ask": "Prepare a Bell pair on two qubits: half the shots read 00 and half 11, never 01 or 10. Use 2 qubits. Measure every qubit at the end into a classical bit of the same index (c = measure q;); outcomes are written with the highest bit first.",
|
|
"qubits": 2,
|
|
"qasm": "OPENQASM 3.0;\ninclude \"stdgates.inc\";\nqubit[2] q;\nbit[2] c;\nh q[0];\ncx q[0], q[1];\nc = measure q;\n",
|
|
"run": {
|
|
"shots": 100
|
|
},
|
|
"intent": [
|
|
{
|
|
"distribution": {
|
|
"00": 0.5,
|
|
"11": 0.5
|
|
}
|
|
}
|
|
],
|
|
"expect": {
|
|
"holds": true,
|
|
"levels": [
|
|
"proof",
|
|
"evidence"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"name": "bell-odd-parity",
|
|
"from": [
|
|
"new: a Bell state of odd parity"
|
|
],
|
|
"covers": [
|
|
"Prepare the Bell state (|01⟩ + |10⟩)/√2 on two qubits: half the shots read 01 and half 10, never 00 or 11."
|
|
],
|
|
"ask": "Prepare the Bell state (|01⟩ + |10⟩)/√2 on two qubits: half the shots read 01 and half 10, never 00 or 11. Use 2 qubits. Measure every qubit at the end into a classical bit of the same index (c = measure q;); outcomes are written with the highest bit first.",
|
|
"qubits": 2,
|
|
"qasm": "OPENQASM 3.0;\nqubit[2] q;\nbit[2] c;\nh q[0];\ncx q[0], q[1];\nx q[0];\nc = measure q;\n",
|
|
"run": {
|
|
"shots": 100
|
|
},
|
|
"intent": [
|
|
{
|
|
"distribution": {
|
|
"01": 0.5,
|
|
"10": 0.5
|
|
}
|
|
}
|
|
],
|
|
"expect": {
|
|
"holds": true,
|
|
"levels": [
|
|
"proof",
|
|
"evidence"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"name": "ghz-3",
|
|
"from": [
|
|
"intents: ghz-prepares-the-chain-s-state, at three qubits"
|
|
],
|
|
"covers": [
|
|
"Prepare a three-qubit GHZ state: half the shots read 000 and half 111."
|
|
],
|
|
"ask": "Prepare a three-qubit GHZ state: half the shots read 000 and half 111. Use 3 qubits. Measure every qubit at the end into a classical bit of the same index (c = measure q;); outcomes are written with the highest bit first.",
|
|
"qubits": 3,
|
|
"qasm": "OPENQASM 3.0;\nqubit[3] q;\nbit[3] c;\nh q[0];\ncx q[0], q[1];\ncx q[1], q[2];\nc = measure q;\n",
|
|
"run": {
|
|
"shots": 100
|
|
},
|
|
"intent": [
|
|
{
|
|
"distribution": {
|
|
"000": 0.5,
|
|
"111": 0.5
|
|
}
|
|
}
|
|
],
|
|
"expect": {
|
|
"holds": true,
|
|
"levels": [
|
|
"proof",
|
|
"evidence"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"name": "ghz-4-prepares",
|
|
"from": [
|
|
"intents: ghz-prepares-the-chain-s-state",
|
|
"examples: ghz"
|
|
],
|
|
"covers": [
|
|
"Prepare the four-qubit GHZ state (|0000⟩ + |1111⟩)/√2 from |0000⟩, in as few layers of gates as you can."
|
|
],
|
|
"ask": "Prepare the four-qubit GHZ state (|0000⟩ + |1111⟩)/√2 from |0000⟩, in as few layers of gates as you can. Use 4 qubits. Measure every qubit at the end into a classical bit of the same index (c = measure q;); outcomes are written with the highest bit first.",
|
|
"qubits": 4,
|
|
"qasm": "OPENQASM 3.0;\ninclude \"stdgates.inc\";\nqubit[4] q;\nbit[4] c;\nh q[0];\ncx q[0], q[1];\ncx q[0], q[2];\ncx q[1], q[3];\nc = measure q;\n",
|
|
"run": {
|
|
"shots": 100
|
|
},
|
|
"intent": [
|
|
{
|
|
"prepares": "OPENQASM 3.0;\ninclude \"stdgates.inc\";\nqubit[4] q;\nbit[4] c;\nh q[0];\ncx q[0], q[1];\ncx q[1], q[2];\ncx q[2], q[3];\nc = measure q;\n",
|
|
"name": "the GHZ state by a chain of CNOTs"
|
|
},
|
|
{
|
|
"distribution": {
|
|
"0000": 0.5,
|
|
"1111": 0.5
|
|
}
|
|
}
|
|
],
|
|
"expect": {
|
|
"holds": true,
|
|
"levels": [
|
|
"proof",
|
|
"evidence"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"name": "uniform-3",
|
|
"from": [
|
|
"new: the uniform superposition"
|
|
],
|
|
"covers": [
|
|
"Put three qubits in the uniform superposition: every one of the eight outcomes equally likely."
|
|
],
|
|
"ask": "Put three qubits in the uniform superposition: every one of the eight outcomes equally likely. Use 3 qubits. Measure every qubit at the end into a classical bit of the same index (c = measure q;); outcomes are written with the highest bit first.",
|
|
"qubits": 3,
|
|
"qasm": "OPENQASM 3.0;\nqubit[3] q;\nbit[3] c;\nh q;\nc = measure q;\n",
|
|
"run": {
|
|
"shots": 100
|
|
},
|
|
"intent": [
|
|
{
|
|
"distribution": {
|
|
"000": 0.125,
|
|
"001": 0.125,
|
|
"010": 0.125,
|
|
"011": 0.125,
|
|
"100": 0.125,
|
|
"101": 0.125,
|
|
"110": 0.125,
|
|
"111": 0.125
|
|
}
|
|
}
|
|
],
|
|
"expect": {
|
|
"holds": true,
|
|
"levels": [
|
|
"proof",
|
|
"evidence"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"name": "grover-2",
|
|
"from": [
|
|
"intents: grover-two-intents",
|
|
"examples: grover"
|
|
],
|
|
"covers": [
|
|
"Run one round of Grover's search over the four basis states of two qubits, marking 11, so that every shot reads 11."
|
|
],
|
|
"ask": "Run one round of Grover's search over the four basis states of two qubits, marking 11, so that every shot reads 11. Use 2 qubits. Measure every qubit at the end into a classical bit of the same index (c = measure q;); outcomes are written with the highest bit first.",
|
|
"qubits": 2,
|
|
"qasm": "OPENQASM 3.0;\ninclude \"stdgates.inc\";\nqubit[2] q;\nbit[2] c;\nh q;\ncz q[0], q[1];\nh q;\nx q;\ncz q[0], q[1];\nx q;\nh q;\nc = measure q;\n",
|
|
"run": {
|
|
"shots": 100
|
|
},
|
|
"intent": [
|
|
{
|
|
"distribution": {
|
|
"11": 1.0
|
|
}
|
|
}
|
|
],
|
|
"expect": {
|
|
"holds": true,
|
|
"levels": [
|
|
"proof",
|
|
"evidence"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"name": "rotation-three-quarters",
|
|
"from": [
|
|
"intents: compiled-first"
|
|
],
|
|
"covers": [
|
|
"Rotate one qubit so that it reads 0 with probability 3/4 and 1 with probability 1/4."
|
|
],
|
|
"ask": "Rotate one qubit so that it reads 0 with probability 3/4 and 1 with probability 1/4. Use 1 qubit. Measure every qubit at the end into a classical bit of the same index (c = measure q;); outcomes are written with the highest bit first.",
|
|
"qubits": 1,
|
|
"qasm": "OPENQASM 3.0;\ninclude \"stdgates.inc\";\nqubit[1] q;\nbit[1] c;\nrx(pi/3) q[0];\nc = measure q;\n",
|
|
"run": {
|
|
"shots": 100
|
|
},
|
|
"intent": [
|
|
{
|
|
"distribution": {
|
|
"0": 0.75,
|
|
"1": 0.25
|
|
}
|
|
}
|
|
],
|
|
"expect": {
|
|
"holds": true,
|
|
"levels": [
|
|
"proof",
|
|
"evidence"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"name": "w-3",
|
|
"from": [
|
|
"new: the W state"
|
|
],
|
|
"covers": [
|
|
"Prepare the three-qubit W state (|001⟩ + |010⟩ + |100⟩)/√3: each of 001, 010 and 100 read a third of the time, and nothing else."
|
|
],
|
|
"ask": "Prepare the three-qubit W state (|001⟩ + |010⟩ + |100⟩)/√3: each of 001, 010 and 100 read a third of the time, and nothing else. Use 3 qubits. Measure every qubit at the end into a classical bit of the same index (c = measure q;); outcomes are written with the highest bit first.",
|
|
"qubits": 3,
|
|
"qasm": "OPENQASM 3.0;\nqubit[3] q;\nbit[3] c;\nry(1.9106332362490186) q[0];\nch q[0], q[1];\ncx q[1], q[2];\ncx q[0], q[1];\nx q[0];\nc = measure q;\n",
|
|
"run": {
|
|
"shots": 100
|
|
},
|
|
"intent": [
|
|
{
|
|
"distribution": {
|
|
"001": 0.3333333333333333,
|
|
"010": 0.3333333333333333,
|
|
"100": 0.3333333333333333
|
|
},
|
|
"tolerance": 1e-09
|
|
}
|
|
],
|
|
"expect": {
|
|
"holds": true,
|
|
"levels": [
|
|
"proof",
|
|
"evidence"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"name": "flip-the-last",
|
|
"from": [
|
|
"new: one qubit flipped"
|
|
],
|
|
"covers": [
|
|
"On three qubits, flip only q[2] to 1, so that every shot reads 100."
|
|
],
|
|
"ask": "On three qubits, flip only q[2] to 1, so that every shot reads 100. Use 3 qubits. Measure every qubit at the end into a classical bit of the same index (c = measure q;); outcomes are written with the highest bit first.",
|
|
"qubits": 3,
|
|
"qasm": "OPENQASM 3.0;\nqubit[3] q;\nbit[3] c;\nx q[2];\nc = measure q;\n",
|
|
"run": {
|
|
"shots": 100
|
|
},
|
|
"intent": [
|
|
{
|
|
"distribution": {
|
|
"100": 1.0
|
|
}
|
|
}
|
|
],
|
|
"expect": {
|
|
"holds": true,
|
|
"levels": [
|
|
"proof",
|
|
"evidence"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"name": "cz-from-cx",
|
|
"from": [
|
|
"intents: the same circuit as a reference (equivalent)"
|
|
],
|
|
"covers": [
|
|
"Write the controlled-Z gate between q[0] and q[1] using only h and cx gates (no cz), with no measurement."
|
|
],
|
|
"ask": "Write the controlled-Z gate between q[0] and q[1] using only h and cx gates (no cz), with no measurement. Use 2 qubits.",
|
|
"qubits": 2,
|
|
"qasm": "OPENQASM 3.0;\nqubit[2] q;\nh q[1];\ncx q[0], q[1];\nh q[1];\n",
|
|
"run": {
|
|
"shots": 100
|
|
},
|
|
"intent": [
|
|
{
|
|
"equivalent": "OPENQASM 3.0;\nqubit[2] q;\ncz q[0], q[1];\n",
|
|
"name": "cz"
|
|
}
|
|
],
|
|
"expect": {
|
|
"holds": true,
|
|
"levels": [
|
|
"proof",
|
|
"evidence"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"name": "swap-from-cx",
|
|
"from": [
|
|
"intents: the same circuit as a reference (equivalent)"
|
|
],
|
|
"covers": [
|
|
"Write a swap of q[0] and q[1] using only cx gates, with no measurement."
|
|
],
|
|
"ask": "Write a swap of q[0] and q[1] using only cx gates, with no measurement. Use 2 qubits.",
|
|
"qubits": 2,
|
|
"qasm": "OPENQASM 3.0;\nqubit[2] q;\ncx q[0], q[1];\ncx q[1], q[0];\ncx q[0], q[1];\n",
|
|
"run": {
|
|
"shots": 100
|
|
},
|
|
"intent": [
|
|
{
|
|
"equivalent": "OPENQASM 3.0;\nqubit[2] q;\nswap q[0], q[1];\n",
|
|
"name": "swap"
|
|
}
|
|
],
|
|
"expect": {
|
|
"holds": true,
|
|
"levels": [
|
|
"proof",
|
|
"evidence"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"name": "merged-rotations",
|
|
"from": [
|
|
"intents: rotations-merge"
|
|
],
|
|
"covers": [
|
|
"Write the same circuit as this one with the two rz gates on q[0] merged into one gate:"
|
|
],
|
|
"ask": "Write the same circuit as this one with the two rz gates on q[0] merged into one gate:\nOPENQASM 3.0;\ninclude \"stdgates.inc\";\nqubit[2] q;\nh q;\nrz(0.3) q[0];\nrz(0.4) q[0];\ncx q[0], q[1];\n Use 2 qubits.",
|
|
"qubits": 2,
|
|
"qasm": "OPENQASM 3.0;\ninclude \"stdgates.inc\";\nqubit[2] q;\nh q;\nrz(0.7) q[0];\ncx q[0], q[1];\n",
|
|
"run": {
|
|
"shots": 100
|
|
},
|
|
"intent": [
|
|
{
|
|
"equivalent": "OPENQASM 3.0;\ninclude \"stdgates.inc\";\nqubit[2] q;\nh q;\nrz(0.3) q[0];\nrz(0.4) q[0];\ncx q[0], q[1];\n",
|
|
"name": "the program with its two rotations"
|
|
}
|
|
],
|
|
"expect": {
|
|
"holds": true,
|
|
"levels": [
|
|
"proof",
|
|
"evidence"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"name": "noisy-bell",
|
|
"from": [
|
|
"examples: noisy-bell"
|
|
],
|
|
"covers": [
|
|
"Prepare a Bell pair on two qubits, to be run under gate errors and readout flips: the engine checks that its noisy distribution is within 0.1 of the ideal pair."
|
|
],
|
|
"ask": "Prepare a Bell pair on two qubits, to be run under gate errors and readout flips: the engine checks that its noisy distribution is within 0.1 of the ideal pair. Use 2 qubits. Measure every qubit at the end into a classical bit of the same index (c = measure q;); outcomes are written with the highest bit first.",
|
|
"qubits": 2,
|
|
"qasm": "OPENQASM 3.0;\ninclude \"stdgates.inc\";\nqubit[2] q;\nbit[2] c;\nh q[0];\ncx q[0], q[1];\nc = measure q;\n",
|
|
"run": {
|
|
"shots": 4000,
|
|
"noise": {
|
|
"depolarizing_1q": 0.01,
|
|
"depolarizing_2q": 0.03,
|
|
"readout": 0.02
|
|
}
|
|
},
|
|
"intent": [
|
|
{
|
|
"distribution": {
|
|
"00": 0.5,
|
|
"11": 0.5
|
|
},
|
|
"tolerance": 0.1
|
|
}
|
|
],
|
|
"expect": {
|
|
"holds": true,
|
|
"levels": [
|
|
"proof",
|
|
"evidence"
|
|
]
|
|
}
|
|
}
|
|
]
|
|
}
|