6.4 KiB
wai-fleet
A WAI fleet (extensions/fleet.md) as a service. An organisation enrols
the nodes it runs by their signing keys: relays, compositors, ingest gateways, room services,
ledgers and audit collectors. Each node contributes the records it already signs, as it signs
them: a relay's session summaries, and every hop's per-hop records with their ledgers, with the
room descriptors that place those in time. The service verifies each record against its node's
enrolled key, keeps it append-only, and for a period signs the fleet report with the
organisation key. The report has one line per node and one per room, with sessions, bytes and
energy, and each line cites the records it came from. Energy keeps its acquisition class: one row
per scope and class, never summed across either. An unmetered figure is counted as unmetered,
never as zero joules, and a total over measured and unmetered figures says it is partial.
Every report is kept with its bundle: the report, its enrolments and its records. Any verifier of the extension checks a bundle without this service, the corpus's second implementation among them:
python3 -I ../live-conformance/fleet_verify.py --bundle week.wai-fleet.json
wai verify week.wai-fleet.json
Accounts, members, sign-on, quotas and billing are not here.
Running it
cargo build --locked --release
WAI_FLEET_DIR=/path/to/fleet WAI_FLEET_KEY=/path/to/organisation.key WAI_FLEET_ADDR=127.0.0.1:8252 \
./target/release/wai-fleet serve
| variable | meaning |
|---|---|
WAI_FLEET_DIR |
the directory the service owns (required); created when absent |
WAI_FLEET_KEY |
the organisation key's file: 32 bytes, the Ed25519 seed; made with mode 0600 when absent |
WAI_FLEET_ADDR |
the address it listens on (default 127.0.0.1:8252) |
wai-fleet key prints the organisation key of WAI_FLEET_KEY. wai-fleet verify FILE prints a
bundle's outcome. Enrolments are signed with the organisation key, by wai enrolment create, and
given to the service; the service signs none itself.
wai-fleet collect URL SOURCE reads a node's summary file and gives each line to the service at
URL as a summary record: SOURCE is a path (a relay's summary.out_path) or an http or
https address that answers with the file. It prints how many lines were held and how many were
refused, by reason. Collecting the same file again holds nothing twice: a line held already is
refused record_duplicate.
The directory:
LOCK held exclusively while a process serves the directory
enrolments.jsonl {"enrolment": hex}, one a line, in the order admitted
records.jsonl each record's carriage, one a line, in the order admitted
reports/<digest>.wfr a report
reports/<digest>.wai-fleet.json its bundle
The two logs are append-only: a line is written whole and the log synced before the request that admitted it is answered, and a line a crash cut short, never answered, is cut off on start. Reports are written whole (to a temporary name, synced, renamed into place). On start the service takes the lock (a second process on the same directory is refused), then reads every enrolment, record and report back and verifies each again, and refuses to start on any that does not.
Endpoints
GET /v1/fleet |
the service, its organisation key and its limits |
GET /v1/fleet/health |
ok |
POST /v1/fleet/enrolments |
{"enrolment": <hex>}: an enrolment the organisation key signed, read and verified, for a node not enrolled yet; 201 with its node, role and digest |
GET /v1/fleet/enrolments |
every enrolment, with its node, role, label, span, digest and the records held of it |
POST /v1/fleet/records |
{"records": [...]}: each record in its carriage, in order, verified against its node's enrolment and every record held; 200 with how many were held and refused, and each one's kind and identity or its reason |
GET /v1/fleet/rooms/{room}/hops |
the per-hop records held of a room, each record and ledger, as a receipt ledger's room source holds them |
POST /v1/fleet/reports |
{"start": <µs>, "end": <µs>}: the report of the period over the enrolments whose span meets it and the records held in it, signed, verified and kept; 201, its digest, its outcome and its bundle |
GET /v1/fleet/reports |
every report kept, with its period |
GET /v1/fleet/reports/{digest} |
a report's bundle and outcome |
POST /v1/fleet/verify |
a bundle: its outcome |
GET /metrics |
the service's counters in the text exposition format (../wai-metrics): requests by route, method and status (wai_fleet_requests_total), records given by outcome (wai_fleet_records_total: held, refused), the enrolments, records and reports held, and the verifications running. No node, room, key or client is a label. It is the operator's, not an operation of the extension: a request that came through a front is answered 404, and a front that sets no forwarding field is configured not to pass the path on |
A refusal is 422 with {"outcome": "refuse", "reason"}, the extension's reason; a record refused
in a batch is named in the batch's answer. A body that is not what the endpoint takes is 400; an
unknown report is 404. A request body is at most 32 MiB and 10 000 records. Requests that change
the fleet run one at a time; POST /v1/fleet/verify verifies at most two bundles at a time.
The service has no accounts. It holds the organisation key because it signs reports. It accepts an enrolment only when the organisation key signed it and a record only when it verifies under an enrolled node's key, whoever submits either. It listens on a loopback address unless told otherwise: who may reach it is decided in front of it.
Checks
cargo test --locked # the corpus's report signed byte for byte, the directory
# recovered after a restart, a line cut short cut off, one
# process per directory, the corpus's summaries read and
# written again by the relay's own types, and records the
# relay's own code signs collected from a file over HTTP
bash smoke.sh http://127.0.0.1:18252 # over HTTP, against an instance started with the corpus's
# organisation key (see .github/workflows/wai-fleet.yml)