open-standards/wai/live-conformance/detection

live-conformance/detection — detection verdicts

The corpus of extensions/detection.md §8: cases of origin claims, robustness records and detection verdicts, accepted and refused, and what a verifier states of each.

Files. vectors.json names the stand-in detector and its definition, and lists the cases, each its name, its provenance, robustness and verdicts (paths of the records given, in order), requireEvidence (the policy evidence of §5.1) and expect. records/ holds every record once, named by its role and the first 16 hexadecimal digits of the SHA-256 of its text.

Equivalence. A conforming verifier given a case's records and policy reaches the outcome expect states (§5): for refuse, the reason of §7, the list and the index; for accept, what §5.3 states: each origin claim's link, subject and class, and each verdict's link, subject, detector, verdict, besideClaim and evidence.

The cases.

• Accepted: verdicts-beside-claims (two verdicts that agree with their claims and two that disagree, the classes as signed), claims-without-verdicts (the same claims, the same classes), and verdict-without-claim.
• One refused case for each reason of §7, some reasons more than once; each name says its defect.

Provenance. wai_detection_vectors gen writes every file but this one from fixed seeds: every key is SHA-256 of a label, and no key is one anyone holds. The detector, wai-detector:conformance, scores a picture by the share of its bright pixels: it detects nothing about how a picture was made and stands for a detector a verifier does not implement. Its robustness records are measured: the generator runs it at each threshold over two bright pictures and two dark ones it draws, under four attacks of robustness-evidence §4. The verdicts' scores are its scores of five pictures the generator draws, named by the content hash of each as a PNG.

Checked by. wai_detection_vectors verify and tests/detection.rs in wai-rs, and ../detection_verify.py, the second implementation, written from the extension's text, which verifies the robustness records with ../robustness_verify.py, requires every mutated copy of an accepted case to be refused and every one of its own checks, dropped, to be caught.