00cfcf0530
`wai-quantum eval run` asks a model each task of a set, once and at temperature 0. The model is asked either for the program alone or for an answer in the answer view layer. The engine then scores what it writes: whether the answer reads, the verdict's level against the task's intent, whether the task is solved, and the joules per solved task where the model server's process energy can be read. Repair rounds send the engine's codes back for a corrected answer, and the report keeps the first try's counts. A model server that is neither on this machine nor on a private network is refused unless allowed. The report ranks nothing, and no report is committed. The task set (wai/quantum-conformance/eval/tasks.json) is fourteen tasks from the intent conformance set and the worked examples. Each has a reference program that a test holds to its intent by proof. `eval corpus` writes each task's answer four ways: the program alone, the layer whole, the layer with the request alone, and a job's JSON. Noise that two calls share is named once. token_cost.py counts each form with a model's own vocabulary. Its report on three open-weight vocabularies is committed beside the corpus. The smoke checks run a scripted model through the harness, and CI names each harness test. Co-Authored-By: Claude Opus 5.5 <[email protected]>
199 lines
12 KiB
YAML
199 lines
12 KiB
YAML
name: wai-quantum command line
|
|
|
|
# The wai-quantum command line (wai/wai-quantum-cli): its tests and clippy with
|
|
# warnings as errors, with and without `submit`; its smoke checks (smoke.sh: lint,
|
|
# plan, run, verify and a forged result refused, key, draw, MCP over stdio, the
|
|
# research tools); and the WASI component, whose output under wasmtime equals the
|
|
# native build's byte for byte. On macOS, also with the GPU state vector. Its tests
|
|
# include the model harness (src/eval.rs): every task's reference solved and an
|
|
# empty program not, a scripted run, a model server on this machine asked and one
|
|
# elsewhere refused, and the corpus (wai/quantum-conformance/eval/) held to the
|
|
# task set.
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
paths:
|
|
- 'wai/wai-quantum/**'
|
|
- 'wai/wai-quantum-cli/**'
|
|
- 'wai/wai-quantum-qpu/**'
|
|
- 'wai/quantum-conformance/eval/**'
|
|
- 'rust-toolchain.toml'
|
|
- '.github/workflows/wai-quantum-cli.yml'
|
|
pull_request:
|
|
paths:
|
|
- 'wai/wai-quantum/**'
|
|
- 'wai/wai-quantum-cli/**'
|
|
- 'wai/wai-quantum-qpu/**'
|
|
- 'wai/quantum-conformance/eval/**'
|
|
- 'rust-toolchain.toml'
|
|
- '.github/workflows/wai-quantum-cli.yml'
|
|
|
|
jobs:
|
|
relevant:
|
|
name: does this change touch it
|
|
runs-on: linux-arm64
|
|
outputs:
|
|
run: ${{ steps.gate.outputs.run }}
|
|
steps:
|
|
- uses: actions/checkout@v5
|
|
with:
|
|
fetch-depth: 0
|
|
- id: gate
|
|
run: |
|
|
set -euo pipefail
|
|
r=$(python3 scripts/pr-touches-paths.py .github/workflows/wai-quantum-cli.yml \
|
|
"${{ github.event.pull_request.base.sha }}" \
|
|
"${{ github.event.pull_request.head.sha }}")
|
|
echo "this change touches the wai-quantum command line: $r"
|
|
echo "run=$r" >> "$GITHUB_OUTPUT"
|
|
|
|
cli-linux:
|
|
name: command line · linux
|
|
needs: relevant
|
|
if: needs.relevant.outputs.run == 'true'
|
|
runs-on: linux
|
|
timeout-minutes: 60
|
|
steps:
|
|
- uses: actions/checkout@v5
|
|
- name: Install Rust 1.98.0 with clippy and the wasm32-wasip2 target
|
|
shell: bash
|
|
run: |
|
|
export PATH="$HOME/.cargo/bin:$PATH"
|
|
command -v rustup >/dev/null 2>&1 || \
|
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain 1.98.0
|
|
rustup toolchain install 1.98.0 --profile minimal --component clippy
|
|
rustup default 1.98.0
|
|
rustup target add --toolchain 1.98.0 wasm32-wasip2
|
|
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
|
|
- name: Tests and clippy, with and without submit
|
|
shell: bash
|
|
working-directory: wai/wai-quantum-cli
|
|
run: |
|
|
set -euo pipefail
|
|
cargo test --locked --release 2>&1 | tee "$RUNNER_TEMP/test.log"
|
|
grep -qE '^test result: ok\. [1-9][0-9]* passed; 0 failed' "$RUNNER_TEMP/test.log" || { echo "::error::no test ran"; exit 1; }
|
|
# Your own providers end to end (tests/providers.rs), against the
|
|
# reference provider this binary serves, each test by name.
|
|
for t in a_person_adds_tests_plans_and_runs_on_a_provider_and_the_receipt_verifies \
|
|
a_run_that_is_not_free_is_refused_until_the_person_confirms_it \
|
|
a_credential_is_kept_in_a_file_only_its_owner_reads_and_appears_in_nothing_printed \
|
|
a_provider_over_http_runs_with_its_credential_in_the_header \
|
|
an_agent_lists_estimates_is_refused_and_runs_only_with_confirm_cost; do
|
|
grep -q "^test $t \.\.\. ok$" "$RUNNER_TEMP/test.log" || { echo "::error::$t did not run and pass"; exit 1; }
|
|
done
|
|
# The model harness and the views in an agent's host, each test by name.
|
|
for t in eval::tests::every_task_s_reference_is_solved_and_an_empty_program_is_not \
|
|
eval::tests::a_scripted_run_reports_what_the_engine_found \
|
|
eval::tests::a_model_on_this_machine_is_asked_and_one_elsewhere_is_refused \
|
|
eval::tests::the_corpus_is_the_task_set_written_three_ways \
|
|
eval::tests::an_answer_is_read_for_its_program_and_scored \
|
|
mcp::tests::a_host_that_draws_views_gets_them_and_one_that_does_not_gets_words; do
|
|
grep -q "^test $t \.\.\. ok$" "$RUNNER_TEMP/test.log" || { echo "::error::$t did not run and pass"; exit 1; }
|
|
done
|
|
# The token counter needs a model's vocabulary to run; here it is only read.
|
|
python3 -I -c 'import ast, sys; ast.parse(open(sys.argv[1]).read())' ../quantum-conformance/eval/token_cost.py
|
|
cargo clippy --locked --release --all-targets -- -D warnings
|
|
cargo clippy --locked --release --all-targets --no-default-features -- -D warnings
|
|
- name: The smoke checks
|
|
shell: bash
|
|
working-directory: wai/wai-quantum-cli
|
|
run: |
|
|
set -euo pipefail
|
|
cargo build --locked --release
|
|
bash smoke.sh ./target/release/wai-quantum
|
|
# The release binary, pinned by version and SHA-256, rather than a build from source.
|
|
- name: Install wasmtime 49.0.2
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
[ "$(uname -m)" = x86_64 ] || { echo "::error::the pinned wasmtime archive is x86_64; this host is $(uname -m)"; exit 1; }
|
|
if ! command -v xz >/dev/null 2>&1; then
|
|
SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo"
|
|
$SUDO apt-get update -qq
|
|
$SUDO apt-get install -y -qq xz-utils
|
|
fi
|
|
v=49.0.2
|
|
f="wasmtime-v$v-x86_64-linux.tar.xz"
|
|
curl --proto '=https' --tlsv1.2 -sSfL -o "$RUNNER_TEMP/$f" \
|
|
"https://github.com/bytecodealliance/wasmtime/releases/download/v$v/$f"
|
|
echo "a4d6e9e3a5a60f527cf7793d674c48930c80c2e8977995b8a275cad3254b9322 $RUNNER_TEMP/$f" | sha256sum -c -
|
|
tar -xJf "$RUNNER_TEMP/$f" -C "$RUNNER_TEMP"
|
|
echo "$RUNNER_TEMP/wasmtime-v$v-x86_64-linux" >> "$GITHUB_PATH"
|
|
# The same source as a WASI component: a circuit's state hash, a run's
|
|
# result and a receipt signed with the published key are the native
|
|
# build's, byte for byte.
|
|
- name: The WASI component gives the native build's bytes
|
|
shell: bash
|
|
working-directory: wai/wai-quantum-cli
|
|
run: |
|
|
set -euo pipefail
|
|
cargo build --locked --release --target wasm32-wasip2 --no-default-features
|
|
w=target/wasm32-wasip2/release/wai-quantum.wasm
|
|
n=target/release/wai-quantum
|
|
printf 'OPENQASM 3.0; qubit[3] q; h q[0]; cx q[0], q[1]; t q[2]; cx q[1], q[2];' > "$RUNNER_TEMP/p.qasm"
|
|
for args in "lab run ghz -n 5" "lab receipt ghz -n 3 --demo-key"; do
|
|
$n $args > "$RUNNER_TEMP/native.txt" 2> /dev/null
|
|
wasmtime run "$w" $args > "$RUNNER_TEMP/wasi.txt" 2> /dev/null
|
|
cmp "$RUNNER_TEMP/native.txt" "$RUNNER_TEMP/wasi.txt" || { echo "::error::$args differs under wasmtime"; diff "$RUNNER_TEMP/native.txt" "$RUNNER_TEMP/wasi.txt" || true; exit 1; }
|
|
done
|
|
$n run --demo-key --json --shots 100 < "$RUNNER_TEMP/p.qasm" 2> /dev/null | python3 -c 'import json,sys; j=json.load(sys.stdin); j["result"].pop("elapsed_ms"); j["result"]["receipt"].pop("digest"); j["result"]["receipt"].pop("signature"); print(json.dumps(j, sort_keys=True))' > "$RUNNER_TEMP/native.json"
|
|
wasmtime run "$w" run --demo-key --json --shots 100 < "$RUNNER_TEMP/p.qasm" 2> /dev/null | python3 -c 'import json,sys; j=json.load(sys.stdin); j["result"].pop("elapsed_ms"); j["result"]["receipt"].pop("digest"); j["result"]["receipt"].pop("signature"); print(json.dumps(j, sort_keys=True))' > "$RUNNER_TEMP/wasi.json"
|
|
cmp "$RUNNER_TEMP/native.json" "$RUNNER_TEMP/wasi.json" || { echo "::error::a run's result differs under wasmtime"; exit 1; }
|
|
# A compile to Clifford+T (rotation synthesis in exact and fixed-point
|
|
# arithmetic, the passes, the certificate and its seal): byte for byte.
|
|
printf 'OPENQASM 3.0; qubit[2] q; h q[0]; rz(0.3) q[0]; cx q[0], q[1]; ry(1.1) q[1]; cp(0.7) q[0], q[1];' > "$RUNNER_TEMP/angles.qasm"
|
|
$n compile --demo-key --json --level 2 --epsilon 1e-8 < "$RUNNER_TEMP/angles.qasm" > "$RUNNER_TEMP/native-compile.json" 2> /dev/null
|
|
wasmtime run "$w" compile --demo-key --json --level 2 --epsilon 1e-8 < "$RUNNER_TEMP/angles.qasm" > "$RUNNER_TEMP/wasi-compile.json" 2> /dev/null
|
|
cmp "$RUNNER_TEMP/native-compile.json" "$RUNNER_TEMP/wasi-compile.json" || { echo "::error::a compile differs under wasmtime"; exit 1; }
|
|
# A run on a device target: the layout search, the noise derived and
|
|
# the density matrix, the same bits.
|
|
for t in synthetic-line-5 synthetic-all-5; do
|
|
$n run --demo-key --json --shots 100 --target $t < "$RUNNER_TEMP/p.qasm" 2> /dev/null | python3 -c 'import json,sys; j=json.load(sys.stdin); j["result"]["receipt"].pop("digest"); j["result"]["receipt"].pop("signature"); print(json.dumps(j, sort_keys=True))' > "$RUNNER_TEMP/native-$t.json"
|
|
wasmtime run "$w" run --demo-key --json --shots 100 --target $t < "$RUNNER_TEMP/p.qasm" 2> /dev/null | python3 -c 'import json,sys; j=json.load(sys.stdin); j["result"]["receipt"].pop("digest"); j["result"]["receipt"].pop("signature"); print(json.dumps(j, sort_keys=True))' > "$RUNNER_TEMP/wasi-$t.json"
|
|
cmp "$RUNNER_TEMP/native-$t.json" "$RUNNER_TEMP/wasi-$t.json" || { echo "::error::a run on $t differs under wasmtime"; exit 1; }
|
|
done
|
|
# A resource estimate (the counts, each rotation's synthesis, the
|
|
# planar model's factory search and the frontier, every receipt and
|
|
# the seal): byte for byte.
|
|
printf 'OPENQASM 3.0; qubit[4] q; h q; ctrl(3) @ x q[0], q[1], q[2], q[3]; rz(0.3) q[0]; ry(1.1) q[2]; t q[1];' > "$RUNNER_TEMP/est.qasm"
|
|
$n estimate --demo-key --json --qubit us-e3 < "$RUNNER_TEMP/est.qasm" > "$RUNNER_TEMP/native-estimate.json" 2> /dev/null
|
|
wasmtime run "$w" estimate --demo-key --json --qubit us-e3 < "$RUNNER_TEMP/est.qasm" > "$RUNNER_TEMP/wasi-estimate.json" 2> /dev/null
|
|
cmp "$RUNNER_TEMP/native-estimate.json" "$RUNNER_TEMP/wasi-estimate.json" || { echo "::error::an estimate differs under wasmtime"; exit 1; }
|
|
echo "the WASI component's output equals the native build's"
|
|
|
|
cli-macos:
|
|
name: command line · macos
|
|
needs: relevant
|
|
if: needs.relevant.outputs.run == 'true'
|
|
runs-on: macos
|
|
timeout-minutes: 60
|
|
steps:
|
|
- uses: actions/checkout@v5
|
|
# launchd's PATH omits cargo on the macOS host runner.
|
|
- name: Put cargo on PATH
|
|
shell: bash
|
|
run: echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
|
|
- name: Tests and clippy, with the GPU state vector
|
|
shell: bash
|
|
working-directory: wai/wai-quantum-cli
|
|
run: |
|
|
set -euo pipefail
|
|
nice -n 10 cargo test --locked --release --features metal 2>&1 | tee "$RUNNER_TEMP/test.log"
|
|
grep -qE '^test result: ok\. [1-9][0-9]* passed; 0 failed' "$RUNNER_TEMP/test.log" || { echo "::error::no test ran"; exit 1; }
|
|
nice -n 10 cargo clippy --locked --release --all-targets --features metal -- -D warnings
|
|
- name: The smoke checks, and a run on the GPU
|
|
shell: bash
|
|
working-directory: wai/wai-quantum-cli
|
|
run: |
|
|
set -euo pipefail
|
|
nice -n 10 cargo build --locked --release --features metal
|
|
bash smoke.sh ./target/release/wai-quantum
|
|
printf 'OPENQASM 3.0; qubit[12] q; bit[12] c; h q; rx(0.3) q[0]; cx q[0], q[1]; c = measure q;' > "$RUNNER_TEMP/gpu.qasm"
|
|
./target/release/wai-quantum run "$RUNNER_TEMP/gpu.qasm" --method statevector_single --shots 200 --demo-key --json > "$RUNNER_TEMP/gpu.json" 2> /dev/null
|
|
python3 -c 'import json,sys; j=json.load(open(sys.argv[1])); r=j["result"]; assert r["accuracy"]["class"]=="single" and sum(r["counts"].values())==200, r' "$RUNNER_TEMP/gpu.json"
|
|
echo "statevector_single ran on $(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["result"]["accuracy"]["device"])' "$RUNNER_TEMP/gpu.json")"
|