# WAI Extension: Conformance Results > Status: Draft. Defines the record `conform-result/1`: the signed outcome of running one of > WAI's conformance corpora against an implementation, whoever wrote it. The record names the > corpus by digest, the implementation by what ran, and each case by its expected and its > actual answer; its summary and its verdict follow from its cases; and it is signed with > Ed25519 over its RFC 8785 form. A verifier checks the record as written, offline: that it is > well formed, that the signature holds, and that the summary and the verdict are what the > cases say. No new transport. The reference implementation is the `conform_result` module of > `wai-rs` (feature `conform_result`), which `wai conform` writes records with and `wai verify` > checks them with; the corpus is > [`conform-result-conformance/`](../conform-result-conformance/), and its second > implementation is `conform-result-conformance/conform_result_verify.py`. > Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174. > > A section number in this text is this extension's unless it names another document. ## 1. Scope In scope: the record (§2), its signing input and signature (§3), what a verifier checks and in what order (§4), and the codes it refuses with (§5). Out of scope: how a run reaches an implementation (a command line, a URL), which corpora and suites exist, and what makes an implementation conformant beyond the run a record states. A record states what one run observed; it is not a certificate. ## 2. The record A conformance result is one JSON text (RFC 8259) in UTF-8: an object with exactly the members below, each of the type given, and no member named twice at any level. Every number in it is an integer from 0 to 2^53 − 1 written in decimal digits, without a sign, a fraction or an exponent. Every other number, and every member not listed, makes the record malformed. | member | type | value | |---|---|---| | `wai` | string | `conform-result/1`, the record's kind and version | | `kind` | string | the kind of implementation run: `verifier`, `decoder`, `packager`, `origin`, or another kind a later revision names | | `suite` | string | the suite that ran | | `standard` | string | the version of WAI whose corpus ran (`v0.4.0`) | | `toolchain` | object | the program that ran the corpus: `name` and `version`, each a string | | `corpus` | object | `dir`, the corpus's directory under `wai/` (a string); `files`, the number of corpus files the run read (an integer); `digest`, the corpus digest of §2.1 | | `implementation` | object | `adapter` (`command` or `http`); `impl`, the command line or the URL (a string); `name` and `version`, what the implementer calls it (each a string or `null`); `files`, an array of objects with exactly `path` (a string) and `sha256` (64 lowercase hexadecimal digits), the files the command names (§2.2) | | `started_at` | string | when the run started, as `YYYY-MM-DDTHH:MM:SSZ`: RFC 3339 §5.6 in UTC, to the second, with month 01–12, day 01–31, hour 00–23, minute 00–59 and second 00–60 | | `cases` | array | one object a case, with exactly `name` (a string), `pass` (a boolean), `expected` and `got` (each a string: what the corpus states, and what the implementation answered) | | `summary` | object | exactly `cases`, `passed` and `failed`, each an integer | | `conformant` | boolean | whether the run passed | | `key` | string | `ephemeral`, for a key made for the run, or `supplied`, for a key the runner holds (§6) | | `signer_pubkey` | string | the Ed25519 public key that signed the record, 64 lowercase hexadecimal digits | | `sig` | string | its signature, 128 lowercase hexadecimal digits (§3) | A writer MUST set `summary.cases` to the number of cases, `summary.passed` to the number whose `pass` is `true` and `summary.failed` to the number whose `pass` is `false`; and MUST set `conformant` to `true` exactly when there is at least one case and every case passed. ### 2.1 The corpus digest The corpus digest is `sha256:` followed by 64 lowercase hexadecimal digits: the SHA-256 of the concatenation of one line for every corpus file the run read, each line the file's SHA-256 in lowercase hexadecimal, two spaces, the file's path relative to `wai/` with `/` between its segments, and a line feed; the lines sorted in ascending byte order, and a line that occurs twice taken once. A party that holds the corpus can compute the digest and compare it; a record made against another corpus has another digest. ### 2.2 The implementation For the `command` adapter, `files` holds the program the command line runs, as found on the runner's `PATH`, and every argument of the command line that is a file, each with the SHA-256 of its bytes when the run started. For the `http` adapter it is empty, and `impl` is the URL. These name what ran on the runner's machine; they are statements of the runner, covered by the signature like every other member. ## 3. The signature The signing input is the 18 bytes `wai:conform-result` followed by the byte `0x01`, followed by the record without its `sig` member, serialised as RFC 8785 specifies (its members sorted, no insignificant whitespace). `signer_pubkey` is part of the signed record. `sig` is a pure Ed25519 signature (RFC 8032 §5.1) over the signing input by the key `signer_pubkey`. It is verified by WAI's strict rule ([SPEC](../SPEC.md) §7 *Signatures*): the key canonical and not of small order, `S` < L, `R` not of small order, and the cofactorless equation compared byte for byte. ## 4. Verification A verifier checks a record in this order and refuses it at the first step that fails, with the code that step names (§5): 1. **It reads.** The bytes are one JSON text in UTF-8 whose value is an object, and no object in it names a member twice. Otherwise `malformed`. 2. **It is this format.** The member `wai` is the string `conform-result/1`. Otherwise `unknown_format`. 3. **Its members.** It has exactly the members of §2, each of the type and form §2 gives. Otherwise `malformed`. 4. **Its signature.** `sig` verifies over the signing input of §3 under `signer_pubkey`, by the strict rule. Otherwise `signature_invalid`. 5. **Its summary.** `summary.cases`, `summary.passed` and `summary.failed` are what its cases count (§2). Otherwise `summary_mismatch`. 6. **Its verdict.** `conformant` is `true` exactly when there is at least one case and every case passed. Otherwise `conformant_mismatch`. A record that passes every step is accepted, and its verdict is its `conformant`. A verifier writes its outcome as `accept conformant=true`, `accept conformant=false`, or `reject step= reason=`. A record that says its run was not conformant is a genuine record of that run, and is accepted as one. ## 5. Reasons The codes this extension defines, each with the step that raises it: | reason | raised at | |---|---| | `malformed` | §4 steps 1 and 3 | | `unknown_format` | §4 step 2 | | `signature_invalid` | §4 step 4 | | `summary_mismatch` | §4 step 5 | | `conformant_mismatch` | §4 step 6 | ## 6. What a signature shows A record signed with a key made for its run (`"key": "ephemeral"`) shows that the record is the one the run wrote: no case, count, digest or name was changed after it was signed. It does not say who ran it. A record signed with a key the runner holds (`"key": "supplied"`) says, in addition, that the holder of that key ran it; which party that is, a relying party learns as it learns any key, for instance from a key log (jwp-receipts §7). Neither makes the implementation's other behaviour conform, and neither makes the run's statements true: a runner can run a corpus other than the one it names, and the digest of §2.1 is how a party holding the corpus checks which one ran. ## 7. Conformance [`conform-result-conformance/`](../conform-result-conformance/) holds records signed with a published test key: accepted ones (conformant, not conformant, with no cases, from either adapter and either kind of key) and refused ones (a case's outcome changed after signing, the corpus digest changed, the summary alone changed, a signature changed in one bit, an `S` past L, a key of small order, a summary that disagrees with the cases though signed (each of its three counts wrong alone, and two at once), a verdict that disagrees with them, an unknown version, a member missing, a member added, a member named twice, a count with a fraction, a digest out of form, bytes that are not JSON), each with the outcome §4 gives it. `wai_conform_result_vectors verify` (in `wai-rs`, feature `conform_result`) and the independent `conform_result_verify.py`, which shares no code with `wai-rs`, reach every vector's outcome. ## 8. Security and privacy considerations - **What a record reveals.** The command line and the paths of the files it names are the runner's, and can name a user or a machine; the URL names where the implementation ran. A runner SHOULD run from paths and addresses it is willing to publish with the record. - **Accepted is not trusted.** Acceptance shows the record is intact and consistent. Whether the corpus it names is WAI's, and whether its signer is a party to rely on, a relying party decides from the digest (§2.1) and from what it knows of the key (§6). - **Bounded reading.** A verifier reads one JSON object and its members once. It MAY refuse a record larger than it is prepared to read, as `malformed`.