open-standards/wai/extensions/conformance-results.md

# WAI Extension: Conformance Results

> Status: Draft. Defines the record `conform-result/1`: the signed outcome of running one of
> WAI's conformance corpora against an implementation, whoever wrote it. The record names the
> corpus by digest, the implementation by what ran, and each case by its expected and its
> actual answer; its summary and its verdict follow from its cases; and it is signed with
> Ed25519 over its RFC 8785 form. A verifier checks the record as written, offline: that it is
> well formed, that the signature holds, and that the summary and the verdict are what the
> cases say. No new transport. The reference implementation is the `conform_result` module of
> `wai-rs` (feature `conform_result`), which `wai conform` writes records with and `wai verify`
> checks them with; the corpus is
> [`conform-result-conformance/`](../conform-result-conformance/), and its second
> implementation is `conform-result-conformance/conform_result_verify.py`.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.
>
> A section number in this text is this extension's unless it names another document.

## 1. Scope

In scope: the record (§2), its signing input and signature (§3), what a verifier checks and in
what order (§4), and the codes it refuses with (§5).

Out of scope: how a run reaches an implementation (a command line, a URL), which corpora and
suites exist, and what makes an implementation conformant beyond the run a record states. A
record states what one run observed; it is not a certificate.

## 2. The record

A conformance result is one JSON text (RFC 8259) in UTF-8: an object with exactly the members
below, each of the type given, and no member named twice at any level. Every number in it is an
integer from 0 to 2^53 − 1 written in decimal digits, without a sign, a fraction or an exponent.
Every other number, and every member not listed, makes the record malformed.

| member | type | value |
|---|---|---|
| `wai` | string | `conform-result/1`, the record's kind and version |
| `kind` | string | the kind of implementation run: `verifier`, `decoder`, `packager`, `origin`, or another kind a later revision names |
| `suite` | string | the suite that ran |
| `standard` | string | the version of WAI whose corpus ran (`v0.4.0`) |
| `toolchain` | object | the program that ran the corpus: `name` and `version`, each a string |
| `corpus` | object | `dir`, the corpus's directory under `wai/` (a string); `files`, the number of corpus files the run read (an integer); `digest`, the corpus digest of §2.1 |
| `implementation` | object | `adapter` (`command` or `http`); `impl`, the command line or the URL (a string); `name` and `version`, what the implementer calls it (each a string or `null`); `files`, an array of objects with exactly `path` (a string) and `sha256` (64 lowercase hexadecimal digits), the files the command names (§2.2) |
| `started_at` | string | when the run started, as `YYYY-MM-DDTHH:MM:SSZ`: RFC 3339 §5.6 in UTC, to the second, with month 01–12, day 01–31, hour 00–23, minute 00–59 and second 00–60 |
| `cases` | array | one object a case, with exactly `name` (a string), `pass` (a boolean), `expected` and `got` (each a string: what the corpus states, and what the implementation answered) |
| `summary` | object | exactly `cases`, `passed` and `failed`, each an integer |
| `conformant` | boolean | whether the run passed |
| `key` | string | `ephemeral`, for a key made for the run, or `supplied`, for a key the runner holds (§6) |
| `signer_pubkey` | string | the Ed25519 public key that signed the record, 64 lowercase hexadecimal digits |
| `sig` | string | its signature, 128 lowercase hexadecimal digits (§3) |

A writer MUST set `summary.cases` to the number of cases, `summary.passed` to the number whose
`pass` is `true` and `summary.failed` to the number whose `pass` is `false`; and MUST set
`conformant` to `true` exactly when there is at least one case and every case passed.

### 2.1 The corpus digest

The corpus digest is `sha256:` followed by 64 lowercase hexadecimal digits: the SHA-256 of the
concatenation of one line for every corpus file the run read, each line the file's SHA-256 in
lowercase hexadecimal, two spaces, the file's path relative to `wai/` with `/` between its
segments, and a line feed; the lines sorted in ascending byte order, and a line that occurs twice
taken once. A party that holds the corpus can compute the digest and compare it; a record made
against another corpus has another digest.

### 2.2 The implementation

For the `command` adapter, `files` holds the program the command line runs, as found on the
runner's `PATH`, and every argument of the command line that is a file, each with the SHA-256 of
its bytes when the run started. For the `http` adapter it is empty, and `impl` is the URL. These
name what ran on the runner's machine; they are statements of the runner, covered by the
signature like every other member.

## 3. The signature

The signing input is the 18 bytes `wai:conform-result` followed by the byte `0x01`, followed by
the record without its `sig` member, serialised as RFC 8785 specifies (its members sorted, no
insignificant whitespace). `signer_pubkey` is part of the signed record.

`sig` is a pure Ed25519 signature (RFC 8032 §5.1) over the signing input by the key
`signer_pubkey`. It is verified by WAI's strict rule ([SPEC](../SPEC.md) §7 *Signatures*):
the key canonical and not of small order, `S` < L, `R` not of small order, and the cofactorless
equation compared byte for byte.

## 4. Verification

A verifier checks a record in this order and refuses it at the first step that fails, with the
code that step names (§5):

1. **It reads.** The bytes are one JSON text in UTF-8 whose value is an object, and no object in
   it names a member twice. Otherwise `malformed`.
2. **It is this format.** The member `wai` is the string `conform-result/1`. Otherwise
   `unknown_format`.
3. **Its members.** It has exactly the members of §2, each of the type and form §2 gives.
   Otherwise `malformed`.
4. **Its signature.** `sig` verifies over the signing input of §3 under `signer_pubkey`, by the
   strict rule. Otherwise `signature_invalid`.
5. **Its summary.** `summary.cases`, `summary.passed` and `summary.failed` are what its cases
   count (§2). Otherwise `summary_mismatch`.
6. **Its verdict.** `conformant` is `true` exactly when there is at least one case and every case
   passed. Otherwise `conformant_mismatch`.

A record that passes every step is accepted, and its verdict is its `conformant`. A verifier
writes its outcome as `accept conformant=true`, `accept conformant=false`, or
`reject step=<n> reason=<code>`. A record that says its run was not conformant is a genuine
record of that run, and is accepted as one.

## 5. Reasons

The codes this extension defines, each with the step that raises it:

| reason | raised at |
|---|---|
| `malformed` | §4 steps 1 and 3 |
| `unknown_format` | §4 step 2 |
| `signature_invalid` | §4 step 4 |
| `summary_mismatch` | §4 step 5 |
| `conformant_mismatch` | §4 step 6 |

## 6. What a signature shows

A record signed with a key made for its run (`"key": "ephemeral"`) shows that the record is the
one the run wrote: no case, count, digest or name was changed after it was signed. It does not
say who ran it. A record signed with a key the runner holds (`"key": "supplied"`) says, in
addition, that the holder of that key ran it; which party that is, a relying party learns as it
learns any key, for instance from a key log (jwp-receipts §7). Neither makes the implementation's
other behaviour conform, and neither makes the run's statements true: a runner can run a corpus
other than the one it names, and the digest of §2.1 is how a party holding the corpus checks
which one ran.

## 7. Conformance

[`conform-result-conformance/`](../conform-result-conformance/) holds records signed with a
published test key: accepted ones (conformant, not conformant, with no cases, from either
adapter and either kind of key) and refused ones (a case's outcome changed after signing, the
corpus digest changed, the summary alone changed, a signature changed in one bit, an `S` past
L, a key of small order, a summary that disagrees with the cases though signed (each of its
three counts wrong alone, and two at once), a verdict that disagrees with them, an unknown
version, a member missing, a member added, a member named twice, a count with a fraction, a
digest out of form, bytes that are not JSON), each with the outcome §4 gives it.
`wai_conform_result_vectors verify` (in `wai-rs`, feature `conform_result`) and the independent
`conform_result_verify.py`, which shares no code with `wai-rs`, reach every vector's outcome.

## 8. Security and privacy considerations

- **What a record reveals.** The command line and the paths of the files it names are the
  runner's, and can name a user or a machine; the URL names where the implementation ran. A
  runner SHOULD run from paths and addresses it is willing to publish with the record.
- **Accepted is not trusted.** Acceptance shows the record is intact and consistent. Whether
  the corpus it names is WAI's, and whether its signer is a party to rely on, a relying party
  decides from the digest (§2.1) and from what it knows of the key (§6).
- **Bounded reading.** A verifier reads one JSON object and its members once. It MAY refuse a
  record larger than it is prepared to read, as `malformed`.