open-standards/wai/live-conformance/robustness

live-conformance/robustness — robustness evidence for soft bindings

The corpus of extensions/robustness-evidence.md §13:

• corpus/<name>.json: the corpus documents (§3), each the JCS text of its document, so the SHA-256 of the file is the document's digest: conformance, three pictures of 256 × 256 whose samples this directory holds; tent64-spectral, tent64-leaves and tent64-clips, the measurement sets of synthesis §7.5.8, by their samples' digests.
• frames/<SHA-256>.rgb: the samples of the conformance pictures, raw RGB8, each file named by the SHA-256 of its bytes.
• records/<name>.json: the signed records, each its JCS text: conformance.json, the record over the conformance corpus; conformance-claim.json, the claim of what that record shows survived; tent64-spectral.json, tent64-leaves.json and tent64-clips.json, the records of wai-mark:tent64 over the measurement sets (§10); and tent64-claim.json, its claim, which is synthesis §7.5.8's list.
• vectors.json: algorithm, the term and digest of wai-mark:tent64; harness, the term of §6; and cases, each with its policy (evidence: reproduced or stated), the algorithms the verifier implements, its records in order as JSON texts, the corpora it was handed with them (record, the record's position, and a file of this directory or a text), the samples it holds (each a SHA-256, the file frames/<SHA-256>.rgb), and what a verifier gives it.

Robustness equivalence. For each case, a conforming verifier reaches the outcome it states:

• refuse: the reason of §12 and the record, the position of the record refused;
• accept: the statements of §9.3, one per record, exactly as written.

A verifier that reaches another outcome, reason, record or statement does not conform.

Provenance. wai_robustness_vectors gen writes every file from fixed seeds: the conformance pictures are two dead-leaves pictures (synthesis_mark::leaves, seeds 200 and 201) and a spectral one (wai.generate.spectral's generator with its committed model, the prompt "a lighthouse at dusk", seed 200), each 256 × 256; the measurement sets are the synthesis corpus's (live-conformance/synthesis-mark/); every key is SHA-256 of wai-robustness corpus key and an index; Ed25519 signing is deterministic. The records' tables are the harness's (§6), run by the generator. No picture comes from outside the repository.

The records of wai-mark:tent64 are reproduced by the generator, which makes the measurement sets; their samples, 25 MB, are not in this directory, so a verifier that does not make them reads those records as stated, and checks their tables against ../synthesis-mark/battery.json, which wai_synthesis_mark_vectors wrote: the same counts, measured by another program.

From wai/wai-rs, the reference regenerates and checks the corpus; from this directory's parent, the second implementation checks it:

cargo run --release --no-default-features --features robustness --bin wai_robustness_vectors -- verify ../live-conformance/robustness
python3 -I robustness_verify.py robustness