open-standards/wai/wai-fleet

wai-fleet

A WAI fleet (extensions/fleet.md) as a service. An organisation enrols the nodes it runs by their signing keys: relays, compositors, ingest gateways, room services, ledgers and audit collectors. Each node contributes the records it already signs, as it signs them: a relay's session summaries, and every hop's per-hop records with their ledgers, with the room descriptors that place those in time. The service verifies each record against its node's enrolled key, keeps it append-only, and for a period signs the fleet report with the organisation key. The report has one line per node and one per room, with sessions, bytes and energy, and each line cites the records it came from. Energy keeps its acquisition class: one row per scope and class, never summed across either. An unmetered figure is counted as unmetered, never as zero joules, and a total over measured and unmetered figures says it is partial.

Every report is kept with its bundle: the report, its enrolments and its records. Any verifier of the extension checks a bundle without this service, the corpus's second implementation among them:

python3 -I ../live-conformance/fleet_verify.py --bundle week.wai-fleet.json
wai verify week.wai-fleet.json

Accounts, members, sign-on, quotas and billing are not here.

Running it

cargo build --locked --release
WAI_FLEET_DIR=/path/to/fleet WAI_FLEET_KEY=/path/to/organisation.key WAI_FLEET_ADDR=127.0.0.1:8252 \
  ./target/release/wai-fleet serve

variable	meaning
WAI_FLEET_DIR	the directory the service owns (required); created when absent
WAI_FLEET_KEY	the organisation key's file: 32 bytes, the Ed25519 seed; made with mode 0600 when absent
WAI_FLEET_ADDR	the address it listens on (default 127.0.0.1:8252)

wai-fleet key prints the organisation key of WAI_FLEET_KEY. wai-fleet verify FILE prints a bundle's outcome. Enrolments are signed with the organisation key, by wai enrolment create, and given to the service; the service signs none itself.

wai-fleet collect URL SOURCE reads a node's summary file and gives each line to the service at URL as a summary record: SOURCE is a path (a relay's summary.out_path) or an http or https address that answers with the file. It prints how many lines were held and how many were refused, by reason. Collecting the same file again holds nothing twice: a line held already is refused record_duplicate.

The directory:

LOCK                                  held exclusively while a process serves the directory
enrolments.jsonl                      {"enrolment": hex}, one a line, in the order admitted
records.jsonl                         each record's carriage, one a line, in the order admitted
reports/<digest>.wfr                  a report
reports/<digest>.wai-fleet.json       its bundle

The two logs are append-only: a line is written whole and the log synced before the request that admitted it is answered, and a line a crash cut short, never answered, is cut off on start. Reports are written whole (to a temporary name, synced, renamed into place). On start the service takes the lock (a second process on the same directory is refused), then reads every enrolment, record and report back and verifies each again, and refuses to start on any that does not.

Endpoints

GET /v1/fleet	the service, its organisation key and its limits
GET /v1/fleet/health	ok
POST /v1/fleet/enrolments	{"enrolment": <hex>}: an enrolment the organisation key signed, read and verified, for a node not enrolled yet; 201 with its node, role and digest
GET /v1/fleet/enrolments	every enrolment, with its node, role, label, span, digest and the records held of it
POST /v1/fleet/records	{"records": [...]}: each record in its carriage, in order, verified against its node's enrolment and every record held; 200 with how many were held and refused, and each one's kind and identity or its reason
GET /v1/fleet/rooms/{room}/hops	the per-hop records held of a room, each record and ledger, as a receipt ledger's room source holds them
POST /v1/fleet/reports	{"start": <µs>, "end": <µs>}: the report of the period over the enrolments whose span meets it and the records held in it, signed, verified and kept; 201, its digest, its outcome and its bundle
GET /v1/fleet/reports	every report kept, with its period
GET /v1/fleet/reports/{digest}	a report's bundle and outcome
POST /v1/fleet/verify	a bundle: its outcome

A refusal is 422 with {"outcome": "refuse", "reason"}, the extension's reason; a record refused in a batch is named in the batch's answer. A body that is not what the endpoint takes is 400; an unknown report is 404. A request body is at most 32 MiB and 10 000 records. Requests that change the fleet run one at a time; POST /v1/fleet/verify verifies at most two bundles at a time.

The service has no accounts. It holds the organisation key because it signs reports. It accepts an enrolment only when the organisation key signed it and a record only when it verifies under an enrolled node's key, whoever submits either. It listens on a loopback address unless told otherwise: who may reach it is decided in front of it.

Checks

cargo test --locked                    # the corpus's report signed byte for byte, the directory
                                       # recovered after a restart, a line cut short cut off, one
                                       # process per directory, the corpus's summaries read and
                                       # written again by the relay's own types, and records the
                                       # relay's own code signs collected from a file over HTTP
bash smoke.sh http://127.0.0.1:18252   # over HTTP, against an instance started with the corpus's
                                       # organisation key (see .github/workflows/wai-fleet.yml)