open-standards/wai/extensions/conformance-results.md

WAI Extension: Conformance Results

Status: Draft. Defines the record conform-result/1: the signed outcome of running one of WAI's conformance corpora against an implementation, whoever wrote it. The record names the corpus by digest, the implementation by what ran, and each case by its expected and its actual answer; its summary and its verdict follow from its cases; and it is signed with Ed25519 over its RFC 8785 form. A verifier checks the record as written, offline: that it is well formed, that the signature holds, and that the summary and the verdict are what the cases say. No new transport. The reference implementation is the conform_result module of wai-rs (feature conform_result), which wai conform writes records with and wai verify checks them with; the corpus is conform-result-conformance/, and its second implementation is conform-result-conformance/conform_result_verify.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

A section number in this text is this extension's unless it names another document.

1. Scope

In scope: the record (§2), its signing input and signature (§3), what a verifier checks and in what order (§4), and the codes it refuses with (§5).

Out of scope: how a run reaches an implementation (a command line, a URL), which corpora and suites exist, and what makes an implementation conformant beyond the run a record states. A record states what one run observed; it is not a certificate.

2. The record

A conformance result is one JSON text (RFC 8259) in UTF-8: an object with exactly the members below, each of the type given, and no member named twice at any level. Every number in it is an integer from 0 to 2^53 − 1 written in decimal digits, without a sign, a fraction or an exponent. Every other number, and every member not listed, makes the record malformed.

member	type	value
wai	string	conform-result/1, the record's kind and version
kind	string	the kind of implementation run: verifier, decoder, packager, origin, or another kind a later revision names
suite	string	the suite that ran
standard	string	the version of WAI whose corpus ran (v0.4.0)
toolchain	object	the program that ran the corpus: name and version, each a string
corpus	object	dir, the corpus's directory under wai/ (a string); files, the number of corpus files the run read (an integer); digest, the corpus digest of §2.1
implementation	object	adapter (command or http); impl, the command line or the URL (a string); name and version, what the implementer calls it (each a string or null); files, an array of objects with exactly path (a string) and sha256 (64 lowercase hexadecimal digits), the files the command names (§2.2)
started_at	string	when the run started, as YYYY-MM-DDTHH:MM:SSZ: RFC 3339 §5.6 in UTC, to the second, with month 01–12, day 01–31, hour 00–23, minute 00–59 and second 00–60
cases	array	one object a case, with exactly name (a string), pass (a boolean), expected and got (each a string: what the corpus states, and what the implementation answered)
summary	object	exactly cases, passed and failed, each an integer
conformant	boolean	whether the run passed
key	string	ephemeral, for a key made for the run, or supplied, for a key the runner holds (§6)
signer_pubkey	string	the Ed25519 public key that signed the record, 64 lowercase hexadecimal digits
sig	string	its signature, 128 lowercase hexadecimal digits (§3)

A writer MUST set summary.cases to the number of cases, summary.passed to the number whose pass is true and summary.failed to the number whose pass is false; and MUST set conformant to true exactly when there is at least one case and every case passed.

2.1 The corpus digest

The corpus digest is sha256: followed by 64 lowercase hexadecimal digits: the SHA-256 of the concatenation of one line for every corpus file the run read, each line the file's SHA-256 in lowercase hexadecimal, two spaces, the file's path relative to wai/ with / between its segments, and a line feed; the lines sorted in ascending byte order, and a line that occurs twice taken once. A party that holds the corpus can compute the digest and compare it; a record made against another corpus has another digest.

2.2 The implementation

For the command adapter, files holds the program the command line runs, as found on the runner's PATH, and every argument of the command line that is a file, each with the SHA-256 of its bytes when the run started. For the http adapter it is empty, and impl is the URL. These name what ran on the runner's machine; they are statements of the runner, covered by the signature like every other member.

3. The signature

The signing input is the 18 bytes wai:conform-result followed by the byte 0x01, followed by the record without its sig member, serialised as RFC 8785 specifies (its members sorted, no insignificant whitespace). signer_pubkey is part of the signed record.

sig is a pure Ed25519 signature (RFC 8032 §5.1) over the signing input by the key signer_pubkey. It is verified by WAI's strict rule (SPEC §7 Signatures): the key canonical and not of small order, S < L, R not of small order, and the cofactorless equation compared byte for byte.

4. Verification

A verifier checks a record in this order and refuses it at the first step that fails, with the code that step names (§5):

1. It reads. The bytes are one JSON text in UTF-8 whose value is an object, and no object in it names a member twice. Otherwise malformed.
2. It is this format. The member wai is the string conform-result/1. Otherwise unknown_format.
3. Its members. It has exactly the members of §2, each of the type and form §2 gives. Otherwise malformed.
4. Its signature. sig verifies over the signing input of §3 under signer_pubkey, by the strict rule. Otherwise signature_invalid.
5. Its summary. summary.cases, summary.passed and summary.failed are what its cases count (§2). Otherwise summary_mismatch.
6. Its verdict. conformant is true exactly when there is at least one case and every case passed. Otherwise conformant_mismatch.

A record that passes every step is accepted, and its verdict is its conformant. A verifier writes its outcome as accept conformant=true, accept conformant=false, or reject step=<n> reason=<code>. A record that says its run was not conformant is a genuine record of that run, and is accepted as one.

5. Reasons

The codes this extension defines, each with the step that raises it:

reason	raised at
malformed	§4 steps 1 and 3
unknown_format	§4 step 2
signature_invalid	§4 step 4
summary_mismatch	§4 step 5
conformant_mismatch	§4 step 6

6. What a signature shows

A record signed with a key made for its run ("key": "ephemeral") shows that the record is the one the run wrote: no case, count, digest or name was changed after it was signed. It does not say who ran it. A record signed with a key the runner holds ("key": "supplied") says, in addition, that the holder of that key ran it; which party that is, a relying party learns as it learns any key, for instance from a key log (jwp-receipts §7). Neither makes the implementation's other behaviour conform, and neither makes the run's statements true: a runner can run a corpus other than the one it names, and the digest of §2.1 is how a party holding the corpus checks which one ran.

7. Conformance

conform-result-conformance/ holds records signed with a published test key: accepted ones (conformant, not conformant, with no cases, from either adapter and either kind of key) and refused ones (a case's outcome changed after signing, the corpus digest changed, the summary alone changed, a signature changed in one bit, an S past L, a key of small order, a summary that disagrees with the cases though signed (each of its three counts wrong alone, and two at once), a verdict that disagrees with them, an unknown version, a member missing, a member added, a member named twice, a count with a fraction, a digest out of form, bytes that are not JSON), each with the outcome §4 gives it. wai_conform_result_vectors verify (in wai-rs, feature conform_result) and the independent conform_result_verify.py, which shares no code with wai-rs, reach every vector's outcome.

8. Security and privacy considerations

• What a record reveals. The command line and the paths of the files it names are the runner's, and can name a user or a machine; the URL names where the implementation ran. A runner SHOULD run from paths and addresses it is willing to publish with the record.
• Accepted is not trusted. Acceptance shows the record is intact and consistent. Whether the corpus it names is WAI's, and whether its signer is a party to rely on, a relying party decides from the digest (§2.1) and from what it knows of the key (§6).
• Bounded reading. A verifier reads one JSON object and its members once. It MAY refuse a record larger than it is prepared to read, as malformed.