open-standards/wai/wai-quantum-service/docs/self-host.md

# Running the quantum job service yourself

The job service at `https://wai.transaction.science/quantum-jobs` is free and takes no account and no key. You can also run the same service on your own machine or in your own network, for a lab, a course, or work that should not leave your premises. This page says how: the binary or the image, its configuration, where it keeps jobs, its keys, and how to put it behind a reverse proxy.

## Run it

From a checkout of the repository, with the toolchain `rust-toolchain.toml` names:

```sh norun=server
cd wai/wai-quantum-service
cargo build --locked --release
./target/release/wai-quantum-service serve          # 127.0.0.1:8241
bash smoke.sh http://127.0.0.1:8241                  # its promises, checked over HTTP
```

Or as an OCI image, built from the repository's root ([`deploy/Containerfile`](../deploy/Containerfile)), whose ignore file keeps the context to the crates the service is built from:

```sh norun=server
docker build -f wai/wai-quantum-service/deploy/Containerfile -t wai-quantum-service .
docker run -d --name wai-quantum -p 127.0.0.1:8241:8241 \
  -v wai-quantum:/var/lib/wai-quantum-service \
  --read-only --tmpfs /tmp --cap-drop ALL --security-opt no-new-privileges \
  wai-quantum-service
```

`podman` takes the same commands. The image's build stage and run stage are each pinned by digest, the build uses the locked dependencies with paths remapped and the timestamp fixed, and the run stage holds the binary alone, run as an unprivileged user. `bash wai/wai-quantum-service/deploy/check-image.sh` builds the image, runs it, holds it to the smoke checks, restarts it to check that it keeps its jobs, and removes everything it made. It pushes nothing.

Each job runs in a child process of the same binary, at a lower scheduling priority than the host's other work, watched for its memory and its time. On macOS the child also runs in a sandbox with no network, no file writes and no other programs. Elsewhere, run the service in a container (as above) or under a service manager that confines it: it needs to listen on its port and to write its store, and nothing else.

## Configure it

Everything is set by environment variables.

| variable | default | meaning |
|---|---|---|
| `WAI_QSVC_ADDR` | `127.0.0.1:8241` | the address it listens on (`0.0.0.0:8241` in the image) |
| `WAI_QSVC_PUBLIC_BASE` | the public service's | the URL it is reached at, for the links it hands out |
| `WAI_QSVC_CONSOLE` | the public console | the page a browser at `/` is sent to |
| `WAI_QSVC_STORE` | `~/.local/share/wai-quantum-service/store` | the job store's directory, or `memory` to keep jobs only while the process runs |
| `WAI_QSVC_RETAIN_SECS` | 86400 | how long a finished job is kept |
| `WAI_QSVC_KEY` | `~/.config/wai-quantum-service/signing.key` | the receipt key, created on first start |
| `WAI_QSVC_RECEIPT_TIME` | `clock` | `clock`: every receipt seals when it was issued, by this host's clock; `none`: receipts seal no time |
| `WAI_QSVC_RECEIPT_VALID_S` | unset | a lifetime in seconds, sealed in every receipt as the moment it is no longer offered |
| `WAI_QSVC_API_KEYS` | unset (no keys) | a file of API keys: set it and every request needs one |
| `WAI_QSVC_TRUSTED_PROXIES` | `127.0.0.1,::1` | peers whose `X-Forwarded-For` names the client |
| `WAI_QSVC_ENERGY` | 1 | 0 meters no job's energy |
| `WAI_QSVC_BUDGET_GB` | 96 | the memory running jobs share |
| `WAI_QSVC_JOB_GB` | 72 | the memory one job may take |
| `WAI_QSVC_WORKERS` | 4 | jobs running at once |
| `WAI_QSVC_MIN_FREE_PCT` | 15 | the share of the host's memory a starting job must leave free |
| `WAI_QSVC_JOB_SECS` | 900 | the longest a job (or a whole set) may run |
| `WAI_QSVC_CLIENT_ACTIVE` | 3 | queued or running jobs one client may hold |
| `WAI_QSVC_CLIENT_GB` | the job size | the memory one client's jobs may hold together |
| `WAI_QSVC_CLIENT_BURST` | 30 | submissions one client may make in a window |
| `WAI_QSVC_READ_GB` | 4 | the memory one request's reading may take (polled; see the README's reading section) |
| `WAI_QSVC_READ_SECS` | 45 | the time one request's reading may take |
| `WAI_QSVC_READ_ANSWER_MIB` | 128 | the most one request's reading may give back to the service, once read |
| `WAI_QSVC_QEC_SECS` | 25 | the time an error-correction call's shots may take, measured as they run |
| `WAI_QSVC_ANSWER_SECS` | 55 | the longest any answer takes, its waits and readings included |
| `WAI_QSVC_CLIENT_KEPT_MIB` | 128 | what one client's ended jobs may keep in memory |
| `WAI_QSVC_KEPT_MIB` | 4096 | what all ended jobs whose results have been read may keep in memory |
| `WAI_QSVC_CLIENT_UNREAD_MIB` | 256 | what one client's results not yet read may hold, each of its running jobs counted at its result's bound and the next at `WAI_QSVC_RESULT_MIB`; past it, its next job is refused (`429`) |
| `WAI_QSVC_UNREAD_MIB` | 8192 | what all results not yet read may hold; past it, every next job is refused (`503`) |
| `WAI_QSVC_RESULT_MIB` | 64 | the most one job's or set's result may take, as served: a request whose result's bound passes it is refused before it runs (`422`, `J031`) |
| `WAI_QSVC_CLIENT_STREAMS` | 4 | long polls (`?wait=`) and event streams one client may hold at once |
| `WAI_QSVC_STREAMS` | 512 | those of all clients, at most a quarter of the connections held |
| `WAI_QSVC_KEPT_MIN_SECS` | 600 | how long an ended job not yet read is kept whatever the two caps above say |
| `WAI_QSVC_CLIENT_VERIFIES` | 120 | receipts one client may have checked in its window (`POST /v1/verify`, `verify_quantum_receipt`) |
| `WAI_QSVC_CLIENT_REQUESTS` | 32 | requests one client may have open at once; past it, `503` before the body is read |
| `WAI_QSVC_CONNECTIONS` | 4096 | connections held at once, at most; lowered to what the open-file limit leaves room for |
| `WAI_QSVC_CLIENT_READS` | 2 | how many of one client's jobs, dry runs and verdicts are read at once (more wait) |
| `WAI_QSVC_CLIENT_CHECKS` | 1 | how many of one client's checks run at once (more wait) |
| `WAI_QSVC_HEADER_SECS` | 150 | how long a client may take to send a request's header section, and how long an idle connection stays open |
| `WAI_QSVC_BODY_SECS` | 30 | how long a client may take to send a request's body |
| `WAI_QSVC_CLIENT_WINDOW_SECS` | 600 | the window |
| `WAI_QSVC_MAX_QUEUE` | 256 | circuit jobs waiting at once, and writes waiting at once, each counted alone |
| `WAI_QSVC_SANDBOX` | 1 | 0 runs the macOS worker and writer unsandboxed, for debugging |
| `WAI_QSVC_MODEL` | unset (no writes) | the model's directory, its files and `wai-model.json` (`wai-quantum-writer manifest DIR` writes it): set it and the service writes programs for tasks (`POST /v1/write`). The directory must hold exactly the model files the manifest lists, and the writer reads every byte against it each time it starts, before it loads the model |
| `WAI_QSVC_WRITER_GB` | 96 | the memory the writer process may hold, its model's mapped pages among it |
| `WAI_QSVC_WRITE_SECS` | 3600 | the longest one write may run |

Size the memory to the host: `WAI_QSVC_BUDGET_GB` below the host's memory, `WAI_QSVC_JOB_GB` at or below it. `GET /v1` states every limit in force, and a dry run (`"dry_run": true`) states the ones a job meets.

## The job store

Every job is written to the store before the request that admitted it is answered, and again at every change of its status. Programs, requests, results and receipts are kept content-addressed, each under the BLAKE3 digest of its bytes: a program's text, or the RFC 8785 canonical text of a request, a result without its receipt, or a receipt. A job's report names them under `digests`.

```text
<store>/LOCK                    held while a process serves the store
<store>/client.key              the key client names are hashed with
<store>/objects/<ab>/<digest>   each object, named by its digest
<store>/jobs/<id>.json          each job's record
```

- **Writes are whole.** Every file is written under a temporary name, synced, renamed into place, and its directory synced, so a crash leaves the old file or the new one. A second process on the same store is refused.
- **A restart loses nothing it answered.** On start the service reloads the store. A finished job is served as it was, and its receipt still verifies. A job that was running is run again, once, from its stored request: it is queued again in its order, and its `interruptions` say when the service stopped. Stopped again while it runs once more, it comes back `interrupted`, with the reason; its request is kept, and sent again it runs to the same result. A queued job is queued again, in its order. A worker whose service has gone stops itself.
- **Objects are checked.** An object is checked against its digest when it is read, and a record that does not read is set aside, not served.
- **Retention.** A finished job is kept for `WAI_QSVC_RETAIN_SECS`, then dropped with every object no other job names.
- **No addresses.** A record names its client by a keyed hash of the client's address, so it can count the client's limits after a restart and says nothing else.
- **Backups.** Copy the store's directory while the service is stopped, or, while it runs, copy `jobs/` first and `objects/` after it: an object is never changed once written, and a record names only objects written before it.

Another kind of store is anything that implements the `Store` trait in [`src/store.rs`](../src/store.rs): keep an object by its digest, read it back checked, keep and list the records, and sweep what no record names.

## The receipt keys

Every result is sealed with the service's Ed25519 key, and the receipt names that key (`key`, in hex). The key set, every key the service has signed with, is published at `GET /.well-known/wai-quantum-receipt-keys`. It sits beside the key file as `<key>.keys.json`.

```json
{ "format": "wai.quantum.receipt-keys/1", "algorithm": "ed25519",
  "keys": [
    { "key": "9f…", "did": "did:key:z6Mk…", "not_before_ms": null, "not_after_ms": 1793000000000,
      "successor": "4c…", "current": false,
      "transition": { "statement": { "format": "wai.quantum.key-transition/1", "retiring": "9f…", "successor": "4c…", "at_ms": 1793000000000 },
                      "retiring_signature": "…", "successor_signature": "…" } },
    { "key": "4c…", "did": "did:key:z6Mk…", "not_before_ms": 1793000000000, "current": true } ] }
```

To rotate the key, run `wai-quantum-service rotate-key` with the same `WAI_QSVC_KEY`, then restart the service. The rotation does four things:

- It retires the current key. Its entry stays in the set with its window closed (`not_after_ms`), so receipts it signed still verify and can be placed in time.
- It makes a new key, whose window opens at the same moment.
- It writes a transition statement naming both keys and the moment, signed by each under the domain `wai:quantum-key-transition` (version 1). Whoever trusted the old key can check that the new one came from its holder.
- It moves the old private key aside to `<key>.retired-<ms>`. The service never reads that file again, and you may destroy it.

`not_before_ms` is null for a key that signed before the set was kept. A key file replaced by hand is taken as current at the next start, and the old key is retired with no transition, which its entry says. The service refuses to start on a set whose recorded transition does not verify.

Every receipt seals when it was issued (`time.issued_ms`, by this host's clock, inside the digest), so a verifier holding the set places a receipt in its key's window from the receipt itself: `wai-quantum verify job.json --trust keys.json`, with the set as it is published. A receipt sealed before receipts stated a time, or with `WAI_QSVC_RECEIPT_TIME=none`, can be placed in no window that has ended; its job's `finished_ms` is the host's unsealed word.

To stop vouching for a key, revoke it:

```sh norun=usage
wai-quantum-service revoke-key 9f… 1793000000000
# {"key": "9f…", …, "revoked_from_ms": 1793000000000}
```

The key's entry gains `revoked_from_ms` (now, when no time is given), and the set is published with it from the next start. A verifier holding the set then refuses that key's receipts issued from that time on, and those that seal no time; what it signed before stands. A key someone else may have used is revoked from the moment it was last known safe, or from 0: whoever holds a key can seal any time with it, so only the revocation's own time decides. Asked again, a key stays revoked from the earliest time asked. The key the service signs with is not revoked while it signs: rotate first. A revocation reaches a reader when they fetch the set again.

## API keys, for a private deployment

The public service takes no key, and nothing changes for it. A private deployment can require one. Make a key for each holder:

```sh norun=usage
wai-quantum-service api-key lab-scheduler
# {"name": "lab-scheduler", "key": "3b9…", "line": "lab-scheduler 7f0…", …}
```

Hand `key` to its holder, and add `line` to the file `WAI_QSVC_API_KEYS` names. The file holds each key's BLAKE3 digest, never the key. With the variable set:

- every request but the open ones carries `Authorization: Bearer <key>`, or is refused with `401` and `J033`. The open ones are `/`, `/v1`, `/v1/health`, `/v1/methods`, `/v1/openapi.json` and the receipt keys.
- a client's limits are counted by its key's name, not its address, so one gateway or scheduler is one client wherever it connects from.

The command line sends a key with `--api-key` or `WAI_QUANTUM_API_KEY`.

## Energy

Each finished job carries its energy inside the signed result: a counter reading with its class, its scope and its declared uncertainty, or `{"unmetered": "<why>"}`, never zero. `GET /v1` says how this host meters (`energy`).

- **macOS:** the kernel's count of the worker process's own CPU energy (`attribution: process`).
- **Linux:** the processor packages' energy counters, read through the kernel's power-capping interface before the worker starts and after it exits. They count whole packages, so the figure is the packages' energy over the job's window, every process on the host included, with no attribution to the job (`attribution: none`). It names how many other jobs ran in the window (`concurrent_jobs`). The counter is used only when every package zone reads and its count is live; its declared uncertainty covers the two reads and the counter's update edges, measured at start.
- **Permissions:** many Linux hosts let only root, or a group such as `energy`, read the count. Add the service's user to that group. In a container the interface is usually hidden; a container that should meter needs it visible and readable, which widens what the container sees of the host. Without it, jobs say they are unmetered and why.

## Open files

Each connection takes a file. The service raises its soft open-file limit to its hard one as it starts and logs the result (`open files: …`), and holds connections under it with room for its readers' and workers' pipes and its store's files (`connections: at most …`). On macOS a launchd service inherits `launchctl limit maxfiles`, soft 256 and hard unlimited by default: the service raises its soft limit itself (to 10240, or `kern.maxfilesperproc` if lower) and holds `WAI_QSVC_CONNECTIONS` (4096). Only where the hard limit is low does the cap fall: at a hard limit of 256 it holds 192 connections, so a plist that sets `HardResourceLimits` (`NumberOfFiles`) should set it to 10240 or more. Whatever the cap, a few connections past it are held for health checks alone, and long polls and event streams take at most a quarter of it.

## Behind a reverse proxy

Run the service on a loopback address and let the proxy take the public traffic.

- **The client's address.** The proxy sets `X-Forwarded-For`, and the service trusts it only from `WAI_QSVC_TRUSTED_PROXIES`. Have the proxy set the header from the connection itself, not pass on what the request carried, or a client could choose its own limits.
- **A path prefix.** Mount the service under a prefix (the public one is `/quantum-jobs`) by stripping the prefix before passing the request on. Set `WAI_QSVC_PUBLIC_BASE` to the prefixed URL, so the links the service hands out are right. A well-known path belongs at the origin's root: map `/.well-known/wai-quantum-receipt-keys` there to the service's own.
- **Event streams.** `GET /v1/jobs/{id}/events` is a server-sent event stream. The proxy must pass it through unbuffered (the service sends `X-Accel-Buffering: no` and a comment every 15 s), and allow it to stay open as long as `WAI_QSVC_JOB_SECS`. Polls with `?wait=` are held up to 25 s.
- **Bodies.** A request body is at most 1 MiB, and `POST /v1/run` waits up to 25 s before answering 202. An answer's body can be as large as a result may be as served, `WAI_QSVC_RESULT_MIB` (64 MiB), with the job's request beside it: a proxy that buffers or caps answers must allow that. A request whose result could be larger is refused before it runs (`422`, `J031`), its bound in the reason. A job `POST /v1/verify` and `/mcp` check is at most 4 MiB and 2^19 values (`too_large` past them), and an MCP message that carries one at most 24 MiB and 64 KiB; a body past 1 MiB is read only while fewer than four such are being read, one a client (`503` past them), and each check holds some 50 MiB in a reading child, on one of its client's check turns.
- **Rate-limit fields.** Pass the `RateLimit-Policy`, `RateLimit` and `Retry-After` headers through.
- **Timeouts.** Every answer comes within `WAI_QSVC_ANSWER_SECS` (55) of the request's arrival, so within a proxy's 60 s wait for an answer's header. The worst case of each route, from arrival: a check, a dry run or a submission waits at most 10 s for a turn and reads for at most `WAI_QSVC_READ_SECS` (45), 55 s in all; `POST /v1/run` and the MCP run tools then wait for their job only for what is left of the 55 s (less 5 s, on MCP, for its verdict) and answer with the job, still running, to poll; an MCP answer's verdict waits for a turn and is read only within what is left, and says where no time was left; `tasks/result` holds a reply for what is left of the 55 s, 50 s at most. With a shorter wait in the proxy, lower `WAI_QSVC_ANSWER_SECS` below it. Keep the proxy's idle connections shorter than `WAI_QSVC_HEADER_SECS` (150), which also bounds an idle connection on the service's side: the proxy then closes its idle connections itself, and never sends a request on one as the service closes it.
- **One address, many people.** The service counts a client by the address the proxy gives it. People behind one address share one client's turns (`WAI_QSVC_CLIENT_READS`, `WAI_QSVC_CLIENT_CHECKS`) and window; raise those, or give such a group an API key.

## Updating it

Readers and workers start from the service's own binary as it was started: on Linux its running image (`/proc/<pid>/exe`), elsewhere a copy kept beside the signing key (`bin/wai-quantum-service-<digest>`). So:

1. Put the new binary in place: a copy over the old one, or a copy beside it renamed over it (on Linux a running binary cannot be written, so rename). The running service goes on reading and running jobs with its own binary.
2. Restart the service. It keeps a copy of the new binary on its first start (on Linux, nothing to keep).
3. Check `GET /v1/health`, then `smoke.sh`. Copies of earlier binaries beside the key are not removed; delete them once no service started from them runs.

A minimal site, in one common proxy's configuration format:

```text
quantum.example.org {
    handle_path /quantum-jobs/* {
        reverse_proxy 127.0.0.1:8241 {
            header_up X-Forwarded-For {remote_host}
            flush_interval -1
            transport http {
                keepalive 60s
            }
        }
    }
}
```

## Check it

```sh norun=account
bash wai/wai-quantum-service/smoke.sh https://quantum.example.org/quantum-jobs
wai-quantum submit prog.qasm --service https://quantum.example.org/quantum-jobs --dry-run
```

The command line's `submit`, `task` and `verify` work against any instance; `verify` checks a receipt offline against the key the receipt names, and the key set says when that key signed.