open-standards/wai/wai-quantum-service/docs/self-host.md

Running the quantum job service yourself

The job service at https://wai.transaction.science/quantum-jobs is free and takes no account and no key. You can also run the same service on your own machine or in your own network, for a lab, a course, or work that should not leave your premises. This page says how: the binary or the image, its configuration, where it keeps jobs, its keys, and how to put it behind a reverse proxy.

Run it

From a checkout of the repository, with the toolchain rust-toolchain.toml names:

cd wai/wai-quantum-service
cargo build --locked --release
./target/release/wai-quantum-service serve          # 127.0.0.1:8241
bash smoke.sh http://127.0.0.1:8241                  # its promises, checked over HTTP

Or as an OCI image, built from the repository's root (deploy/Containerfile), whose ignore file keeps the context to the crates the service is built from:

docker build -f wai/wai-quantum-service/deploy/Containerfile -t wai-quantum-service .
docker run -d --name wai-quantum -p 127.0.0.1:8241:8241 \
  -v wai-quantum:/var/lib/wai-quantum-service \
  --read-only --tmpfs /tmp --cap-drop ALL --security-opt no-new-privileges \
  wai-quantum-service

podman takes the same commands. The image's build stage and run stage are each pinned by digest, the build uses the locked dependencies with paths remapped and the timestamp fixed, and the run stage holds the binary alone, run as an unprivileged user. bash wai/wai-quantum-service/deploy/check-image.sh builds the image, runs it, holds it to the smoke checks, restarts it to check that it keeps its jobs, and removes everything it made. It pushes nothing.

Each job runs in a child process of the same binary, at a lower scheduling priority than the host's other work, watched for its memory and its time. On macOS the child also runs in a sandbox with no network, no file writes and no other programs. Elsewhere, run the service in a container (as above) or under a service manager that confines it: it needs to listen on its port and to write its store, and nothing else.

Configure it

Everything is set by environment variables.

variable	default	meaning
WAI_QSVC_ADDR	127.0.0.1:8241	the address it listens on (0.0.0.0:8241 in the image)
WAI_QSVC_PUBLIC_BASE	the public service's	the URL it is reached at, for the links it hands out
WAI_QSVC_CONSOLE	the public console	the page a browser at / is sent to
WAI_QSVC_STORE	~/.local/share/wai-quantum-service/store	the job store's directory, or memory to keep jobs only while the process runs
WAI_QSVC_RETAIN_SECS	86400	how long a finished job is kept
WAI_QSVC_KEY	~/.config/wai-quantum-service/signing.key	the receipt key, created on first start
WAI_QSVC_API_KEYS	unset (no keys)	a file of API keys: set it and every request needs one
WAI_QSVC_TRUSTED_PROXIES	127.0.0.1,::1	peers whose X-Forwarded-For names the client
WAI_QSVC_ENERGY	1	0 meters no job's energy
WAI_QSVC_BUDGET_GB	96	the memory running jobs share
WAI_QSVC_JOB_GB	72	the memory one job may take
WAI_QSVC_WORKERS	4	jobs running at once
WAI_QSVC_MIN_FREE_PCT	15	the share of the host's memory a starting job must leave free
WAI_QSVC_JOB_SECS	900	the longest a job (or a whole set) may run
WAI_QSVC_CLIENT_ACTIVE	3	queued or running jobs one client may hold
WAI_QSVC_CLIENT_GB	the job size	the memory one client's jobs may hold together
WAI_QSVC_CLIENT_BURST	30	submissions one client may make in a window
WAI_QSVC_READ_GB	4	the memory one request's reading may take (polled; see the README's reading section)
WAI_QSVC_READ_SECS	45	the time one request's reading may take
WAI_QSVC_READ_ANSWER_MIB	128	the most one request's reading may give back to the service, once read
WAI_QSVC_QEC_SECS	25	the time an error-correction call's shots may take, measured as they run
WAI_QSVC_ANSWER_SECS	55	the longest any answer takes, its waits and readings included
WAI_QSVC_CLIENT_KEPT_MIB	128	what one client's ended jobs may keep in memory
WAI_QSVC_KEPT_MIB	4096	what all ended jobs whose results have been read may keep in memory
WAI_QSVC_CLIENT_UNREAD_MIB	256	what one client's results not yet read may hold, each of its running jobs counted at its result's bound and the next at WAI_QSVC_RESULT_MIB; past it, its next job is refused (429)
WAI_QSVC_UNREAD_MIB	8192	what all results not yet read may hold; past it, every next job is refused (503)
WAI_QSVC_RESULT_MIB	64	the most one job's or set's result may take, as served: a request whose result's bound passes it is refused before it runs (422, J031)
WAI_QSVC_CLIENT_STREAMS	4	long polls (?wait=) and event streams one client may hold at once
WAI_QSVC_STREAMS	512	those of all clients, at most a quarter of the connections held
WAI_QSVC_KEPT_MIN_SECS	600	how long an ended job not yet read is kept whatever the two caps above say
WAI_QSVC_CLIENT_VERIFIES	120	receipts one client may have checked in its window (POST /v1/verify, verify_quantum_receipt)
WAI_QSVC_CLIENT_REQUESTS	32	requests one client may have open at once; past it, 503 before the body is read
WAI_QSVC_CONNECTIONS	4096	connections held at once, at most; lowered to what the open-file limit leaves room for
WAI_QSVC_CLIENT_READS	2	how many of one client's jobs, dry runs and verdicts are read at once (more wait)
WAI_QSVC_CLIENT_CHECKS	1	how many of one client's checks run at once (more wait)
WAI_QSVC_HEADER_SECS	150	how long a client may take to send a request's header section, and how long an idle connection stays open
WAI_QSVC_BODY_SECS	30	how long a client may take to send a request's body
WAI_QSVC_CLIENT_WINDOW_SECS	600	the window
WAI_QSVC_MAX_QUEUE	256	circuit jobs waiting at once, and writes waiting at once, each counted alone
WAI_QSVC_SANDBOX	1	0 runs the macOS worker and writer unsandboxed, for debugging
WAI_QSVC_MODEL	unset (no writes)	the model's directory, its files and wai-model.json (wai-quantum-writer manifest DIR writes it): set it and the service writes programs for tasks (POST /v1/write). The directory must hold exactly the model files the manifest lists, and the writer reads every byte against it each time it starts, before it loads the model
WAI_QSVC_WRITER_GB	96	the memory the writer process may hold, its model's mapped pages among it
WAI_QSVC_WRITE_SECS	3600	the longest one write may run

Size the memory to the host: WAI_QSVC_BUDGET_GB below the host's memory, WAI_QSVC_JOB_GB at or below it. GET /v1 states every limit in force, and a dry run ("dry_run": true) states the ones a job meets.

The job store

Every job is written to the store before the request that admitted it is answered, and again at every change of its status. Programs, requests, results and receipts are kept content-addressed, each under the BLAKE3 digest of its bytes: a program's text, or the RFC 8785 canonical text of a request, a result without its receipt, or a receipt. A job's report names them under digests.

<store>/LOCK                    held while a process serves the store
<store>/client.key              the key client names are hashed with
<store>/objects/<ab>/<digest>   each object, named by its digest
<store>/jobs/<id>.json          each job's record

• Writes are whole. Every file is written under a temporary name, synced, renamed into place, and its directory synced, so a crash leaves the old file or the new one. A second process on the same store is refused.
• A restart loses nothing it answered. On start the service reloads the store. A finished job is served as it was, and its receipt still verifies. A job that was running is run again, once, from its stored request: it is queued again in its order, and its interruptions say when the service stopped. Stopped again while it runs once more, it comes back interrupted, with the reason; its request is kept, and sent again it runs to the same result. A queued job is queued again, in its order. A worker whose service has gone stops itself.
• Objects are checked. An object is checked against its digest when it is read, and a record that does not read is set aside, not served.
• Retention. A finished job is kept for WAI_QSVC_RETAIN_SECS, then dropped with every object no other job names.
• No addresses. A record names its client by a keyed hash of the client's address, so it can count the client's limits after a restart and says nothing else.
• Backups. Copy the store's directory while the service is stopped, or, while it runs, copy jobs/ first and objects/ after it: an object is never changed once written, and a record names only objects written before it.

Another kind of store is anything that implements the Store trait in src/store.rs: keep an object by its digest, read it back checked, keep and list the records, and sweep what no record names.

The receipt keys

Every result is sealed with the service's Ed25519 key, and the receipt names that key (key, in hex). The key set, every key the service has signed with, is published at GET /.well-known/wai-quantum-receipt-keys. It sits beside the key file as <key>.keys.json.

{ "format": "wai.quantum.receipt-keys/1", "algorithm": "ed25519",
  "keys": [
    { "key": "9f…", "did": "did:key:z6Mk…", "not_before_ms": null, "not_after_ms": 1793000000000,
      "successor": "4c…", "current": false,
      "transition": { "statement": { "format": "wai.quantum.key-transition/1", "retiring": "9f…", "successor": "4c…", "at_ms": 1793000000000 },
                      "retiring_signature": "…", "successor_signature": "…" } },
    { "key": "4c…", "did": "did:key:z6Mk…", "not_before_ms": 1793000000000, "current": true } ] }

To rotate the key, run wai-quantum-service rotate-key with the same WAI_QSVC_KEY, then restart the service. The rotation does four things:

• It retires the current key. Its entry stays in the set with its window closed (not_after_ms), so receipts it signed still verify and can be placed in time.
• It makes a new key, whose window opens at the same moment.
• It writes a transition statement naming both keys and the moment, signed by each under the domain wai:quantum-key-transition (version 1). Whoever trusted the old key can check that the new one came from its holder.
• It moves the old private key aside to <key>.retired-<ms>. The service never reads that file again, and you may destroy it.

not_before_ms is null for a key that signed before the set was kept. A key file replaced by hand is taken as current at the next start, and the old key is retired with no transition, which its entry says. The service refuses to start on a set whose recorded transition does not verify. A result itself carries no wall-clock time, so a receipt is placed in a key's window by the job's finished_ms or by your own record of when you received it.

API keys, for a private deployment

The public service takes no key, and nothing changes for it. A private deployment can require one. Make a key for each holder:

wai-quantum-service api-key lab-scheduler
# {"name": "lab-scheduler", "key": "3b9…", "line": "lab-scheduler 7f0…", …}

Hand key to its holder, and add line to the file WAI_QSVC_API_KEYS names. The file holds each key's BLAKE3 digest, never the key. With the variable set:

• every request but the open ones carries Authorization: Bearer <key>, or is refused with 401 and J033. The open ones are /, /v1, /v1/health, /v1/methods, /v1/openapi.json and the receipt keys.
• a client's limits are counted by its key's name, not its address, so one gateway or scheduler is one client wherever it connects from.

The command line sends a key with --api-key or WAI_QUANTUM_API_KEY.

Energy

Each finished job carries its energy inside the signed result: a counter reading with its class, its scope and its declared uncertainty, or {"unmetered": "<why>"}, never zero. GET /v1 says how this host meters (energy).

• macOS: the kernel's count of the worker process's own CPU energy (attribution: process).
• Linux: the processor packages' energy counters, read through the kernel's power-capping interface before the worker starts and after it exits. They count whole packages, so the figure is the packages' energy over the job's window, every process on the host included, with no attribution to the job (attribution: none). It names how many other jobs ran in the window (concurrent_jobs). The counter is used only when every package zone reads and its count is live; its declared uncertainty covers the two reads and the counter's update edges, measured at start.
• Permissions: many Linux hosts let only root, or a group such as energy, read the count. Add the service's user to that group. In a container the interface is usually hidden; a container that should meter needs it visible and readable, which widens what the container sees of the host. Without it, jobs say they are unmetered and why.

Open files

Each connection takes a file. The service raises its soft open-file limit to its hard one as it starts and logs the result (open files: …), and holds connections under it with room for its readers' and workers' pipes and its store's files (connections: at most …). On macOS a launchd service inherits launchctl limit maxfiles, soft 256 and hard unlimited by default: the service raises its soft limit itself (to 10240, or kern.maxfilesperproc if lower) and holds WAI_QSVC_CONNECTIONS (4096). Only where the hard limit is low does the cap fall: at a hard limit of 256 it holds 192 connections, so a plist that sets HardResourceLimits (NumberOfFiles) should set it to 10240 or more. Whatever the cap, a few connections past it are held for health checks alone, and long polls and event streams take at most a quarter of it.

Behind a reverse proxy

Run the service on a loopback address and let the proxy take the public traffic.

• The client's address. The proxy sets X-Forwarded-For, and the service trusts it only from WAI_QSVC_TRUSTED_PROXIES. Have the proxy set the header from the connection itself, not pass on what the request carried, or a client could choose its own limits.
• A path prefix. Mount the service under a prefix (the public one is /quantum-jobs) by stripping the prefix before passing the request on. Set WAI_QSVC_PUBLIC_BASE to the prefixed URL, so the links the service hands out are right. A well-known path belongs at the origin's root: map /.well-known/wai-quantum-receipt-keys there to the service's own.
• Event streams. GET /v1/jobs/{id}/events is a server-sent event stream. The proxy must pass it through unbuffered (the service sends X-Accel-Buffering: no and a comment every 15 s), and allow it to stay open as long as WAI_QSVC_JOB_SECS. Polls with ?wait= are held up to 25 s.
• Bodies. A request body is at most 1 MiB, and POST /v1/run waits up to 25 s before answering 202. An answer's body can be as large as a result may be as served, WAI_QSVC_RESULT_MIB (64 MiB), with the job's request beside it: a proxy that buffers or caps answers must allow that. A request whose result could be larger is refused before it runs (422, J031), its bound in the reason. A job POST /v1/verify and /mcp check is at most 4 MiB and 2^19 values (too_large past them), and an MCP message that carries one at most 24 MiB and 64 KiB; a body past 1 MiB is read only while fewer than four such are being read, one a client (503 past them), and each check holds some 50 MiB in a reading child, on one of its client's check turns.
• Rate-limit fields. Pass the RateLimit-Policy, RateLimit and Retry-After headers through.
• Timeouts. Every answer comes within WAI_QSVC_ANSWER_SECS (55) of the request's arrival, so within a proxy's 60 s wait for an answer's header. The worst case of each route, from arrival: a check, a dry run or a submission waits at most 10 s for a turn and reads for at most WAI_QSVC_READ_SECS (45), 55 s in all; POST /v1/run and the MCP run tools then wait for their job only for what is left of the 55 s (less 5 s, on MCP, for its verdict) and answer with the job, still running, to poll; an MCP answer's verdict waits for a turn and is read only within what is left, and says where no time was left; tasks/result holds a reply for what is left of the 55 s, 50 s at most. With a shorter wait in the proxy, lower WAI_QSVC_ANSWER_SECS below it. Keep the proxy's idle connections shorter than WAI_QSVC_HEADER_SECS (150), which also bounds an idle connection on the service's side: the proxy then closes its idle connections itself, and never sends a request on one as the service closes it.
• One address, many people. The service counts a client by the address the proxy gives it. People behind one address share one client's turns (WAI_QSVC_CLIENT_READS, WAI_QSVC_CLIENT_CHECKS) and window; raise those, or give such a group an API key.

Updating it

Readers and workers start from the service's own binary as it was started: on Linux its running image (/proc/<pid>/exe), elsewhere a copy kept beside the signing key (bin/wai-quantum-service-<digest>). So:

1. Put the new binary in place: a copy over the old one, or a copy beside it renamed over it (on Linux a running binary cannot be written, so rename). The running service goes on reading and running jobs with its own binary.
2. Restart the service. It keeps a copy of the new binary on its first start (on Linux, nothing to keep).
3. Check GET /v1/health, then smoke.sh. Copies of earlier binaries beside the key are not removed; delete them once no service started from them runs.

A minimal site, in one common proxy's configuration format:

quantum.example.org {
    handle_path /quantum-jobs/* {
        reverse_proxy 127.0.0.1:8241 {
            header_up X-Forwarded-For {remote_host}
            flush_interval -1
            transport http {
                keepalive 60s
            }
        }
    }
}

Check it

bash wai/wai-quantum-service/smoke.sh https://quantum.example.org/quantum-jobs
wai-quantum submit prog.qasm --service https://quantum.example.org/quantum-jobs --dry-run

The command line's submit, task and verify work against any instance; verify checks a receipt offline against the key the receipt names, and the key set says when that key signed.