open-standards/wai/conform-result-conformance

conform-result-conformance

The conformance corpus of conformance-results (§7): records of conform-result/1, the signed outcome of a conformance run, each with the outcome §4 gives it.

Provenance. Generated here by wai_conform_result_vectors gen, from records written in code and signed with published one-byte seeds: 32 bytes of 0x43 (the key of every record but one) and of 0x44 (a key a runner holds, in valid_http_supplied_key). Both are test keys and attest nothing. Ed25519 signing is deterministic (RFC 8032), so the file is the same bytes on every machine. Every run, corpus file, program, path and URL a record names is an example; no record states a run that took place.

Each vector declares one outcome, set from the extension's text in the generator and never copied from a verifier: accept conformant=true, accept conformant=false, or reject step=<n> reason=<code> at a step of §4 with a code of §5. A verifier that reaches another outcome does not conform.

Files

file	what it is
vectors.json	The corpus: each vector's name, what it is (about), the record's bytes (record, a string, since some vectors are bytes a JSON reader must refuse) and its outcome (expected); and the corpus keys' public keys.
conform_result_verify.py	The second implementation, written from the extension's text and sharing no code with wai-rs. It takes WAI's strict Ed25519 rule from ../claim-conformance/_ed25519.py, which is written from RFC 8032; it reads the JSON and writes the RFC 8785 form itself. Standard library only.

Vectors

• Accepted: a conformant run (valid); a run with a failed case (valid_not_conformant), a genuine record of a run that did not conform; a run with no case (valid_no_cases); and a decoder's run from the HTTP adapter, signed with a runner's key, with a name and version (valid_http_supplied_key).
• Refused at step 4, signature_invalid: a failed case turned into a pass after signing, its summary and verdict made to agree (flipped_case); the corpus digest changed (corpus_digest_changed); the summary alone changed (summary_changed); one bit of the signature changed (bad_signature); the signature's S replaced by S + L (signature_s_not_reduced); the identity as the key with (R, S) = (B, 1), which the cofactorless equation accepts over any message (key_small_order).
• Signed as they are, and refused: a summary that does not count its cases (step 5, summary_mismatch): two counts wrong, and each of the three counts wrong alone with the others right; conformant though a case failed, and conformant with no case (step 6, conformant_mismatch); another version of the format (step 2, unknown_format); a member missing and a member added (step 3, malformed).
• Bytes refused before a signature is read: the verdict named twice (step 1); a count written with a fraction and a digest in upper-case hexadecimal (step 3); bytes that are not a JSON text (step 1).

Checks

From wai/wai-rs:

cargo run --no-default-features --features conform_result --bin wai_conform_result_vectors -- verify ../conform-result-conformance
cargo test --no-default-features --features conform_result --test conform_result

verify requires vectors.json to regenerate byte for byte, no other file but this README and the second implementation, and every vector to reach its outcome. The test adds that every reason of §5 refuses a vector, that §5 lists exactly the reference's codes, and that a record changed anywhere after it is signed is refused.

From wai/:

python3 -I conform-result-conformance/conform_result_verify.py conform-result-conformance
python3 -I conform-result-conformance/conform_result_verify.py --skip <check> conform-result-conformance

The first must report every vector's outcome and that the corpus refuses with every reason of §5. The second leaves out one check of §4 (duplicates, format, members, signature, summary, verdict) and succeeds only when the corpus then fails: each check is guarded by a vector. wai_conform_result_vectors verdicts and conform_result_verify.py verdicts print name<TAB>outcome for any corpus-format file, and wai conform verifier --suite conform-result runs a verifier against this corpus.