Transaction-Science/open-standards
Watch
1
0
Fork
You've already forked open-standards
 0
Code Issues Pull requests Projects Releases 17 Packages Wiki Activity Actions 62

wai conform: JUnit and HTML reports, independent verifiers beside an implementation, the corpus union as one command, and wai doctor #642

Closed
dcharlot wants to merge 4 commits from wai/conform-output into main
pull from: wai/conform-output
merge into: Transaction-Science:main
Conversation 0 Commits 4 Files changed 112 +4581 -452
112 changed files with 4581 additions and 452 deletions
Download patch file Download diff file Expand all files Collapse all files

9
.github/wai-conformance/README.md vendored
View file

@ -16,8 +16,13 @@ a file, or a line to a file of its own, and edits no line of the workflow, unles
run in ascending order, a tie by file name), `features:` (each list compiles alone in
`features-alone`, and every feature joins `WAI_FEATURES`), and where it needs them `alone:`,
`build: pq`, `release-features:`, `bins:`, `pq-bins:`, `paths:` (the extension texts it
reads, which the workflow's gate then runs on) and `working-directory:`. Prose comments
follow the header, after a `#` line.
reads, which the workflow's gate then runs on) and `working-directory:`. Its corpus's
independent verifier is an `independent:` line, `<name> <area> [needs=<packages>]
[verdicts=<file>] [dir=<dir>] python3 [-I] <script> <args>`, the command line its README
gives, run in `wai/` by the job's one step `wai conform --independent` and never by the body;
`wai-rs/build.rs` reads the binary's table of independent verifiers from these lines. A file
with such lines and no body is a declaration: its verifiers run in that step alone. Prose
comments follow the header, after a `#` line.
- `corpora/<name>.txt`: one cargo-class corpus, verified by its own tool on every leg of
`cargo-classes` (`corpus:`, `order:`, `bin:`, `dir:`).
- A test file of `wai/wai-rs/tests/` whose first line is `// wai-conformance: int-kernels`

5
.github/wai-conformance/verifiers/access-services.sh vendored
View file

@ -2,13 +2,15 @@
# order: 160
# features: access_services
# paths: wai/extensions/access-services.md
# independent: access-services live python3 -I live-conformance/access_services_verify.py live-conformance/access
#
# wai.access.services (extensions/access-services.md): the reference module's unit tests
# (words, full and prefix alignment with their walk back, signed integers, gains) and
# tests/access_services.rs (vectors.json and each DAPT document regenerate byte for byte
# from their fixed seeds and no other file is there, wai-rs gives every vector and
# document its stated outcome and figures, every reason of §10 refuses a case, and §10
# lists exactly the reference's codes); then the corpus's second implementation,
# lists exactly the reference's codes); and, in the step of every independent verifier
# above, the corpus's second implementation,
# access_services_verify.py, written from the extension's text and sharing no code with
# wai-rs, which verifies events and captions with timed_text_verify.py and every record
# with WAI's strict Ed25519 rule, computes every figure and writes every DAPT document
@ -22,4 +24,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib access_services::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test access_services)
python3 -I live-conformance/access_services_verify.py live-conformance/access

5
.github/wai-conformance/verifiers/ad-presentation.sh vendored
View file

@ -2,6 +2,7 @@
# order: 250
# features: live_ads
# paths: wai/extensions/ad-presentation.md
# independent: ad-presentation live python3 -I live-conformance/ad_presentation_verify.py live-conformance/presentation
#
# wai.live.ad_presentation (extensions/ad-presentation.md): the reference module's unit
# tests (claims round-trip and their rules refuse, the quartile, the qualifying run, the
@ -9,7 +10,8 @@
# tests/ad_presentation.rs (every file of the committed corpus regenerates byte for byte
# from its fixed seeds and no other file is there, RFC 9578's own token verifies under its
# key, wai-rs gives every room its outcome and every claim its verdict, and §11 lists
# exactly the reference's codes); then the corpus's second implementation,
# exactly the reference's codes); and, in the step of every independent verifier above,
# the corpus's second implementation,
# ad_presentation_verify.py, written from the extension's text and sharing no code with
# wai-rs, which verifies each room with live_ads_verify.py, judges every claim and its
# RSASSA-PSS itself, requires every mutated copy of the corpus to fail and catches each of
@ -24,5 +26,4 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib live_ads::presentation::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test ad_presentation)
python3 -I live-conformance/ad_presentation_verify.py live-conformance/presentation
node --test ../joule-code/joule-code-rs/crates/joule-moq-web/web/ads/presentation.test.mjs

5
.github/wai-conformance/verifiers/age-assurance.sh vendored
View file

@ -2,13 +2,15 @@
# order: 520
# features: age_assurance
# paths: wai/extensions/age-assurance.md
# independent: age-assurance live python3 -I live-conformance/age_assurance_verify.py live-conformance/age
#
# wai.age.assurance (extensions/age-assurance.md): the reference module's unit tests (the
# pinned AlgorithmIdentifier and RFC 9578's first vector through this module, the
# TokenChallenge read exactly, the suppression floor, the grant attribute `age`, a span's
# coverage) and tests/age_assurance.rs (vectors.json regenerates byte for byte from its
# fixed seeds and no other file is there, wai-rs gives every case its stated outcome, and
# §11 lists exactly the reference's codes, each of which the corpus gives); then the
# §11 lists exactly the reference's codes, each of which the corpus gives); and, in the
# step of every independent verifier above, the
# corpus's second implementation, age_assurance_verify.py, written from the extension's
# text and sharing no code with wai-rs, which reads every record, writes JCS and verifies
# every token's RSASSA-PSS itself, reads rooms with live_rooms_verify.py, checks signatures
@ -24,4 +26,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib age_assurance::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test age_assurance)
python3 -I live-conformance/age_assurance_verify.py live-conformance/age

7
.github/wai-conformance/verifiers/ai-usage.sh vendored
View file

@ -2,6 +2,8 @@
# order: 360
# features: ai_usage
# paths: wai/extensions/ai-usage.md
# independent: ai-usage live python3 -I live-conformance/ai_usage_verify.py live-conformance/usage
# independent: ai-usage-claims live needs=cryptography python3 -I claim-conformance/claim_verify.py consumption live-conformance/usage/vectors.json
#
# wai.ai.usage (extensions/ai-usage.md): the reference module's unit tests (RFC 9651's
# Dictionary parser against the vocabulary's examples and the RFC's edges, statements
@ -10,7 +12,8 @@
# tests/ai_usage.rs (vectors.json regenerates byte for byte from its fixed seeds and no
# other file is there, wai-rs gives every statement, object, room, response, catalog,
# claim and gate its stated outcome, §9 lists exactly the reference's codes, each of which
# the corpus gives, and jwp-receipts §3.1 lists the class rules); then the corpus's second
# the corpus gives, and jwp-receipts §3.1 lists the class rules); and, in the step of every
# independent verifier above, the corpus's second
# implementation, ai_usage_verify.py, written from the extension's text and sharing no
# code with wai-rs, which parses Structured Fields, canonical forms, receipts and claims
# itself, reads rooms with live_rooms_verify.py, checks signatures with WAI's strict
@ -28,5 +31,3 @@ set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib ai_usage::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib claim_receipt::consumption::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test ai_usage)
python3 -I live-conformance/ai_usage_verify.py live-conformance/usage
python3 -I claim-conformance/claim_verify.py consumption live-conformance/usage/vectors.json

5
.github/wai-conformance/verifiers/avatar-face.sh vendored
View file

@ -3,6 +3,7 @@
# features: avatar_face
# release-features: avatar_face
# paths: wai/extensions/avatars.md
# independent: avatar-face live python3 -I live-conformance/avatar_face_verify.py live-conformance/avatar-face
#
# wai.avatar.face (extensions/avatars.md §6 to §13): the module's unit tests (the
# generator's wai-ops/1 graph against §7.6's arithmetic written out loop by loop; the warp
@ -10,7 +11,8 @@
# and tests/avatar_face.rs (the committed corpus regenerates byte for byte from its fixed
# seeds and its pinned inputs, and no other file is there; wai-rs gives every form, frame
# and case what it states; §10 lists exactly the reference's codes; §7.9's figures are
# training.json's; the registered generator meets acc64 and not acc32); then the corpus's
# training.json's; the registered generator meets acc64 and not acc32); and, in the step
# of every independent verifier above, the corpus's
# second implementation, avatar_face_verify.py, written from the extension's text and
# sharing no code with wai-rs, which runs the driver and the generator's arithmetic in
# Python's integers, verifies the claims, the log and the label policy with
@ -25,4 +27,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib avatar_face::)
(cd wai-rs && cargo test --release --no-default-features --features "$WAI_RELEASE_FEATURES" --test avatar_face)
python3 -I live-conformance/avatar_face_verify.py live-conformance/avatar-face

7
.github/wai-conformance/verifiers/bearer-receipts.sh vendored
View file

@ -2,6 +2,8 @@
# order: 530
# features: bearer_receipts
# paths: wai/extensions/bearer-receipts.md
# independent: bearer live needs=cryptography python3 -I live-conformance/bearer_verify.py live-conformance/bearers
# independent: bearer-claims live needs=cryptography python3 -I claim-conformance/claim_verify.py bearer live-conformance/bearers/vectors.json
#
# wai.bearer.receipts (extensions/bearer-receipts.md): the reference module's unit tests
# (the bound, the bitmap rules, the bearer classes, and the corpus reaching every stated
@ -10,7 +12,8 @@
# every case its stated outcome, the cases that carry one receipt on every bearer class
# carry the same bytes with the same object verdicts, §10 lists exactly the reference's
# codes, each of which the corpus gives, and jwp-receipts §3.1 lists the class codes at
# their steps); then the corpus's second implementation, bearer_verify.py, written from
# their steps); and, in the step of every independent verifier above, the corpus's second
# implementation, bearer_verify.py, written from
# the extension's text and sharing no code with wai-rs, which verifies groups and objects
# with jwp_verify.py, claims through jwp-receipts §3.1 steps 1-5 with claim_verify.py and
# confirmations with signer_verify.py (each needs the `cryptography` package installed
@ -27,5 +30,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib bearer)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test bearer_receipts)
python3 -I live-conformance/bearer_verify.py live-conformance/bearers
python3 -I claim-conformance/claim_verify.py bearer live-conformance/bearers/vectors.json

5
.github/wai-conformance/verifiers/browser-sink.sh vendored
View file

@ -2,6 +2,7 @@
# order: 470
# features: browser_sink browser_sink,receipt_claims
# paths: wai/extensions/browser-sink.md
# independent: browser-sink live python3 -I live-conformance/browser_sink_verify.py live-conformance/browser
#
# The browser sink (extensions/browser-sink.md): the reference module's unit tests (the
# registry gate's rows, registered tiers, integers as §5 reads them, a binding's
@ -12,7 +13,8 @@
# §11 appears in the corpus and §11 lists exactly the reference's codes, and the chain
# the generator signs by hand verifies under jwp_delivery); wai-live-web's browser
# functions held to the same corpus natively, and its wasm-route decoder verifying,
# decoding and binding every chain object to the published frame digests; then the
# decoding and binding every chain object to the published frame digests; and, in the
# step of every independent verifier above, the
# corpus's second implementation, browser_sink_verify.py, written from the extension's
# text and sharing no code with wai-rs, which reads the registry export for each
# capability's class, recomputes every route, evaluation, energy entry, binding, chain
@ -31,4 +33,3 @@ set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test browser_sink)
(cd wai-rs && "$RUNNER_TEMP/wai-bins/union/wai_browser_sink_vectors" verify ../live-conformance/browser)
(cd wai-live-web && cargo test --locked --features int_video --lib browser_tests)
python3 -I live-conformance/browser_sink_verify.py live-conformance/browser

5
.github/wai-conformance/verifiers/capture-take.sh vendored
View file

@ -2,12 +2,14 @@
# order: 120
# features: capture_take
# paths: wai/extensions/capture-binding.md
# independent: capture-take live python3 -I live-conformance/capture_take_verify.py live-conformance/takes
#
# wai.capture.take (extensions/capture-binding.md): the reference module's unit tests,
# the output tree's (whose RFC 6962 hash and RFC 9162 consistency proofs the take tree
# reuses), and tests/capture_take.rs (the committed corpus regenerates byte for byte
# from its fixed seeds, wai-rs gives every vector its stated outcome, every reason of
# §12 appears in it, and §12 lists exactly the reference's codes); then the corpus's
# §12 appears in it, and §12 lists exactly the reference's codes); and, in the step of
# every independent verifier above, the corpus's
# second implementation, capture_take_verify.py, written from the extension's text and
# sharing no code with wai-rs, which verifies each room with live_rooms_verify.py and
# every signature with WAI's strict Ed25519 (claim-conformance/_ed25519.py), and
@ -20,4 +22,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib -- capture_take:: output_tree::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test capture_take)
python3 -I live-conformance/capture_take_verify.py live-conformance/takes

7
.github/wai-conformance/verifiers/carousel.sh vendored
View file

@ -2,6 +2,8 @@
# order: 540
# features: carousel
# paths: wai/extensions/carousel.md
# independent: carousel live needs=cryptography python3 -I live-conformance/carousel_verify.py live-conformance/carousel
# independent: carousel-claims live needs=cryptography python3 -I claim-conformance/claim_verify.py carousel live-conformance/carousel/vectors.json
#
# wai.bearer.carousel (extensions/carousel.md): the reference module's unit tests (each code's
# bounds and division into source blocks, K', the passes of advance and repeat, plans, and the
@ -11,7 +13,8 @@
# jwp-receipts §3.1 lists the class codes at their steps, K' and Partition[] agree with an
# independent RFC 6330 implementation for every block size, and each RaptorQ object a case
# holds available decodes with that implementation from what a receiver within the case's
# loss bound holds); then the corpus's second implementation, carousel_verify.py, written
# loss bound holds); and, in the step of every independent verifier above, the corpus's second
# implementation, carousel_verify.py, written
# from the extension's text and sharing no code with wai-rs, which verifies groups and
# objects with jwp_verify.py and claims through steps 1-5 with claim_verify.py, writes the
# context checks, the availability and the plans itself, reads §9 and Appendix A, requires
@ -25,5 +28,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib -- carousel:: claim_receipt::schedule::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test carousel)
python3 -I live-conformance/carousel_verify.py live-conformance/carousel
python3 -I claim-conformance/claim_verify.py carousel live-conformance/carousel/vectors.json

2
.github/wai-conformance/verifiers/catalogue-prior.sh vendored
View file

@ -1,5 +1,6 @@
# step: catalogue prior · the record and the text quoting it
# order: 30
# independent: catalogue-prior trust python3 -I tools/wai_catalogue_prior_verify.py research/catalogue-prior research/catalogue-prior.md
#
# The catalogue-prior record (research/catalogue-prior.md): each reading checked against
# the SHA-256 its run.json states; every BD-rate the run stored recomputed from the
@ -8,4 +9,3 @@
# scripts that took them (tools/wai_catalogue_prior_verify.py, standard library only,
# run isolated); and every figure the page quotes checked (each is followed by a `cat`
# marker naming its value).
python3 -I tools/wai_catalogue_prior_verify.py research/catalogue-prior research/catalogue-prior.md

5
.github/wai-conformance/verifiers/channel-lineup.sh vendored
View file

@ -2,6 +2,7 @@
# order: 270
# features: channel_lineup
# paths: wai/extensions/channel-lineup.md
# independent: channel-lineup live needs=cryptography python3 -I live-conformance/channel_lineup_verify.py live-conformance/channels
#
# wai.video.lineup (extensions/channel-lineup.md): the reference module's unit tests (times,
# the document type declaration, the reasons) and tests/channel_lineup.rs (every file of the
@ -9,7 +10,8 @@
# there; wai-rs gives every import and every verification its outcome; §10 lists exactly
# the reference's codes; §9's commands, run as the text writes them through `wai`, yield
# the corpus's receipt, lineup, cues and guides, and a refusal is its reason with exit 1);
# then the corpus's second implementation, channel_lineup_verify.py, written from the
# and, in the step of every independent verifier above, the corpus's second implementation,
# channel_lineup_verify.py, written from the
# extension's text and sharing no code with wai-rs, which reimports every vector, requires
# every outcome and fact, and requires every mutated copy of the corpus to fail and every
# check it drops to be caught.
@ -23,4 +25,3 @@ set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib channel_lineup::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test channel_lineup)
(cd wai-rs && "$RUNNER_TEMP/wai-bins/union/wai_channel_lineup_vectors" verify ../live-conformance/channels)
python3 -I live-conformance/channel_lineup_verify.py live-conformance/channels

17
.github/wai-conformance/verifiers/claims-jwp-signers.sh vendored
View file

@ -3,17 +3,22 @@
# features: key_discovery
# bins: wai_claim_vectors wai_jwp_vectors wai_signer_vectors
# pq-bins: wai_claim_vectors wai_jwp_vectors wai_signer_vectors
# independent: claim trust needs=cryptography verdicts=claim-conformance/vectors.json python3 claim-conformance/claim_verify.py claim-conformance/vectors.json
# independent: jwp trust needs=cryptography verdicts=jwp-conformance/vectors.json python3 jwp-conformance/jwp_verify.py jwp-conformance/vectors.json
# independent: signer trust needs=cryptography verdicts=signer-conformance/vectors.json python3 signer-conformance/signer_verify.py signer-conformance/vectors.json
#
# chain_differential.py signs receipt chains around jwp-receipts §3 step 3 and §6.1
# (two keys, two tracks, both payloads, replays, forks, gaps, the guard and the
# The claim, JWP and key-discovery verifiers (claim_verify.py, jwp_verify.py and
# signer_verify.py) run over their corpora in the step of every independent verifier
# above; here, against wai-rs, beyond the corpora: differential.py gives seeded inputs
# the corpus does not hold to wai_claim_vectors and claim_verify.py and requires the
# same verdict; chain_differential.py signs receipt chains around jwp-receipts §3 step 3
# and §6.1 (two keys, two tracks, both payloads, replays, forks, gaps, the guard and the
# freshness bound), and chain_gap_differential.py genuine chains up to 90 groups
# delivered shuffled, withheld and walked back with rewritten links and forks; both
# require wai_jwp_vectors and jwp_verify.py to give every chain the same verdict. Two
# fixed seeds each, so a run is reproducible.
# fixed seeds each, so a run is reproducible. Last, signer_verify.py's key-discovery
# verdicts equal wai_signer_vectors'.
set -euo pipefail
python3 claim-conformance/claim_verify.py claim-conformance/vectors.json
python3 jwp-conformance/jwp_verify.py jwp-conformance/vectors.json
python3 signer-conformance/signer_verify.py signer-conformance/vectors.json
python3 signer-conformance/signer_verify.py verdicts signer-conformance/vectors.json > "$RUNNER_TEMP/signer-py.tsv"
# Each differential against both builds the first step copied (see its comment).
for build in union pq; do

2
.github/wai-conformance/verifiers/cmsd-read.sh vendored
View file

@ -1,8 +1,8 @@
# step: cmsd_read_verify.py
# order: 750
# independent: cmsd-read delivery python3 -I telemetry-conformance/cmsd_read_verify.py telemetry-conformance
#
# CMSD reading (extensions/delivery-telemetry.md §6.6): the second implementation, written
# from that text and RFC 9651 with its own field parser, reads every field of
# telemetry-conformance/cmsd-read/ and every field expected/cmsd.json states, and requires
# every mutated copy to fail. wai-rs's reading is checked by wai_telemetry verify above.
python3 -I telemetry-conformance/cmsd_read_verify.py telemetry-conformance

5
.github/wai-conformance/verifiers/co-performance.sh vendored
View file

@ -2,13 +2,15 @@
# order: 510
# features: co_performance
# paths: wai/extensions/co-performance.md
# independent: co-performance live python3 -I live-conformance/co_performance_verify.py live-conformance/coperformance
#
# wai.live.coperformance (extensions/co-performance.md): the reference module's unit tests
# (the policy and every statement round-trip and their rules refuse, the attributes'
# decimal integers, a schedule's turns in bars and microseconds, the symbolic bodies read
# whole) and tests/co_performance.rs (vectors.json regenerates byte for byte from its fixed
# seeds and no other file is there, wai-rs gives every room its stated outcome, and §10
# lists exactly the reference's codes, each of which the corpus gives); then the corpus's
# lists exactly the reference's codes, each of which the corpus gives); and, in the step of
# every independent verifier above, the corpus's
# second implementation, co_performance_verify.py, written from the extension's text and
# sharing no code with wai-rs, which verifies each room with live_rooms_verify.py and every
# signature with WAI's strict Ed25519 rule from claim-conformance, recomputes every intent,
@ -24,4 +26,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib co_performance::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test co_performance)
python3 -I live-conformance/co_performance_verify.py live-conformance/coperformance

5
.github/wai-conformance/verifiers/co-watch.sh vendored
View file

@ -2,13 +2,15 @@
# order: 580
# features: co_watch
# paths: wai/extensions/co-watch.md
# independent: co-watch live python3 -I live-conformance/co_watch_verify.py live-conformance/cowatch
#
# wai.live.cowatch (extensions/co-watch.md): the reference module's unit tests (the policy
# and its rules, statements read back and refused, the controlled position in the order of
# §5.4, each branch of the catch-up rule, percentiles by nearest rank, the control attribute)
# and tests/co_watch.rs (vectors.json regenerates byte for byte from its fixed seeds and no
# other file is there, wai-rs gives every room and catch-up case its stated outcome, and §12
# lists exactly the reference's codes, each of which the corpus gives); then the corpus's
# lists exactly the reference's codes, each of which the corpus gives); and, in the step of
# every independent verifier above, the corpus's
# second implementation, co_watch_verify.py, written from the extension's text and sharing
# no code with wai-rs, which verifies each room with live_rooms_verify.py and every
# signature with WAI's strict Ed25519 rule, recomputes the controlled position, every
@ -26,5 +28,4 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib co_watch::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test co_watch)
python3 -I live-conformance/co_watch_verify.py live-conformance/cowatch
node --test ../joule-code/joule-code-rs/crates/joule-moq-web/web/room/cowatch.test.mjs

9
.github/wai-conformance/verifiers/conformance-results.sh vendored
View file

@ -2,17 +2,19 @@
# order: 170
# features: conform_result
# paths: wai/extensions/conformance-results.md
# independent: conform-result trust verdicts=conform-result-conformance/vectors.json python3 -I conform-result-conformance/conform_result_verify.py conform-result-conformance
#
# conform-result/1 (extensions/conformance-results.md): the reference module's unit tests
# (a signed record accepted and every member covered, the strict rule's small-order forgery
# refused, the date form, a vector for every reason) and tests/conform_result.rs (the
# committed corpus regenerates byte for byte, wai-rs gives every vector its stated outcome,
# every reason of §5 refuses a vector, §5 lists exactly the reference's codes, and no leaf
# of a record survives a change after signing); then wai_conform_result_vectors verify, and
# of a record survives a change after signing); then wai_conform_result_vectors verify;
# and, with each check of §4 left out in turn (--skip), the corpus must fail
# the corpus's second implementation, conform_result_verify.py, written from the
# extension's text and sharing no code with wai-rs (WAI's strict Ed25519 rule from
# claim-conformance/_ed25519.py), which reads the extension's reason table; and with each
# check of §4 left out in turn (--skip), the corpus must fail it. Standard library only,
# claim-conformance/_ed25519.py), which reads the extension's reason table and runs whole
# in the step of every independent verifier above. Standard library only,
# run isolated (-I).
#
# The paths this step brings to the workflow's trigger, and why:
@ -23,7 +25,6 @@ set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib conform_result::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test conform_result)
"$RUNNER_TEMP/wai-bins/union/wai_conform_result_vectors" verify conform-result-conformance
python3 -I conform-result-conformance/conform_result_verify.py conform-result-conformance
for c in duplicates format members signature summary verdict; do
python3 -I conform-result-conformance/conform_result_verify.py --skip "$c" conform-result-conformance | tail -1
done

5
.github/wai-conformance/verifiers/contribution.sh vendored
View file

@ -2,6 +2,7 @@
# order: 370
# features: contribution
# paths: wai/extensions/contribution.md
# independent: contribution live python3 -I live-conformance/contribution_verify.py live-conformance/contribution
#
# wai.media.contribution (extensions/contribution.md): the reference module's unit tests
# (every proof of the steps' tree walks to its root, the copying operations follow their
@ -9,7 +10,8 @@
# regenerates byte for byte from its fixed seeds and no other file is there, wai-rs gives
# every case its stated outcome and statements, §11 lists exactly the reference's codes,
# each of which the corpus gives, and the corpus generates with the model file of
# wai.generate.spectral's conformance vector); then the corpus's second implementation,
# wai.generate.spectral's conformance vector); and, in the step of every independent
# verifier above, the corpus's second implementation,
# contribution_verify.py, written from the extension's text and sharing no code with
# wai-rs, which evaluates every pinned operation, runs the generator in its own arithmetic
# after checking it against the generator's conformance vector, computes every map, share
@ -25,4 +27,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib contribution::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test contribution)
python3 -I live-conformance/contribution_verify.py live-conformance/contribution

5
.github/wai-conformance/verifiers/cue-injection-ertmp.sh vendored
View file

@ -1,5 +1,7 @@
# step: cue injection and Enhanced RTMP · injection_verify.py and ertmp_verify.py
# order: 690
# independent: injection live python3 -I live-conformance/injection_verify.py --self-test live-conformance/injection
# independent: ertmp live python3 -I live-conformance/ertmp_verify.py --self-test live-conformance/ertmp
#
# wai.live.injection (extensions/cue-injection.md) and Enhanced RTMP ingest
# (extensions/live-ingest.md §12): both corpora are generated through joule-moq-ingest,
@ -13,6 +15,3 @@
# RTMP message of every session and requires every object of every room track and what
# was not carried. Each then requires every mutated copy of its corpus to fail
# (--self-test). Standard library only, run isolated (-I).
set -euo pipefail
python3 -I live-conformance/injection_verify.py --self-test live-conformance/injection
python3 -I live-conformance/ertmp_verify.py --self-test live-conformance/ertmp

2
.github/wai-conformance/verifiers/delivery-sim.sh vendored
View file

@ -1,7 +1,7 @@
# step: sim_verify.py
# order: 80
# independent: delivery-sim delivery python3 staged-measure-conformance/sim_verify.py staged-measure-conformance
#
# The delivery simulator's second implementation: Appendix B from the text, every case
# of staged-measure-conformance/sim/ reproduced and its claim checked, with a
# counter-case required for each part of the stall bound a schedule can lose.
python3 staged-measure-conformance/sim_verify.py staged-measure-conformance

5
.github/wai-conformance/verifiers/deriving-edge.sh vendored
View file

@ -2,6 +2,7 @@
# order: 680
# features: derive_edge
# paths: wai/extensions/deriving-edge.md
# independent: edge delivery python3 -I edge-conformance/edge_verify.py edge-conformance
#
# wai.edge.derivation (extensions/deriving-edge.md): the reference module's unit tests
# (the operations pixel by pixel, which operations the registry's composition rule makes
@ -12,7 +13,8 @@
# byte for byte from its fixed seeds and no other file is there, wai-rs gives every vector
# its stated outcome, §9 lists exactly the reference's codes and §4 its derivations, and
# the reference edge wai_edge, run twice over a clip, makes a chain of two hops that
# verifies and is refused once a byte changes); then the corpus's second implementation,
# verifies and is refused once a byte changes); and, in the step of every independent
# verifier above, the corpus's second implementation,
# edge_verify.py, written from the extension's text and sharing no code with wai-rs,
# which reads receipts and writes stills itself, re-executes every replicate hop,
# requires every mutated copy of the corpus to fail, and drops each of its own checks in
@ -27,4 +29,3 @@ set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib derive_edge::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib png_still::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test derive_edge)
python3 -I edge-conformance/edge_verify.py edge-conformance

17
.github/wai-conformance/verifiers/determinism-cost.sh vendored
View file

@ -1,15 +1,16 @@
# step: determinism cost · the record, the text quoting it, and wai_det_cost
# order: 20
# features: neural_int
# independent: determinism-cost trust python3 -I tools/wai_determinism_cost_verify.py research/determinism-cost/record.json research/determinism-cost.md extensions/determinism-tiers.md
#
# The determinism-cost record (research/determinism-cost.md): its summary recomputed
# from its readings by a second implementation that shares no code with the tool that
# wrote it (tools/wai_determinism_cost_verify.py, standard library only, run isolated),
# The determinism-cost record (research/determinism-cost.md): wai_det_cost, the decoder
# the record ran, held to the committed int_hyper vector and every WIH1 refusal fixture
# of int-payload-conformance/. The record itself, its summary recomputed from its
# readings by a second implementation that shares no code with the tool that
# wrote it (tools/wai_determinism_cost_verify.py, standard library only, run isolated)
# and every figure the research page and determinism-tiers §7 quote from it checked
# against it (each is followed by a `det` marker naming its place in the record). Then
# wai_det_cost, the decoder the record ran, held to the committed int_hyper vector and
# every WIH1 refusal fixture of int-payload-conformance/.
# against it (each is followed by a `det` marker naming its place in the record), is
# checked in the step of
# every independent verifier above.
set -euo pipefail
python3 -I tools/wai_determinism_cost_verify.py research/determinism-cost/record.json \
research/determinism-cost.md extensions/determinism-tiers.md
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test det_cost_bin)

5
.github/wai-conformance/verifiers/direction.sh vendored
View file

@ -2,6 +2,7 @@
# order: 500
# features: direction
# paths: wai/extensions/direction.md
# independent: direction live python3 -I live-conformance/direction_verify.py live-conformance/direction
#
# wai.media.direction (extensions/direction.md): the reference module's unit tests (the
# interpolations reach both keys and stay between them, the framing measure at its bounds,
@ -10,7 +11,8 @@
# fixed seeds and no other file is there, wai-rs gives every case its stated outcome and
# statements, §13 lists exactly the reference's codes, each of which the corpus gives, and
# the corpus generates with the model file of wai.generate.spectral's conformance vector);
# then the corpus's second implementation, direction_verify.py, written from the
# and, in the step of every independent verifier above, the corpus's second implementation,
# direction_verify.py, written from the
# extension's text and sharing no code with wai-rs, which runs the generator in its own
# arithmetic after checking it against the generator's conformance vector, renders every
# frame, interpolates every track, measures every box and writes every statement and C2PA
@ -26,4 +28,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib direction::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test direction)
python3 -I live-conformance/direction_verify.py live-conformance/direction

2
.github/wai-conformance/verifiers/e2ee.sh vendored
View file

@ -1,6 +1,7 @@
# step: end-to-end encrypted rooms · e2ee_verify.py
# order: 650
# paths: wai/extensions/confidentiality.md
# independent: e2ee live python3 -I live-conformance/e2ee_verify.py live-conformance/e2ee
#
# wai.live.e2ee (extensions/confidentiality.md): the corpus's second implementation,
# e2ee_verify.py, written from the extension's text and the documents it reads and
@ -16,4 +17,3 @@
# The paths this step brings to the workflow's trigger, and why:
# wai/extensions/confidentiality.md: e2ee_verify.py reads this extension's §13 table, and the
# registry's wire identifiers test reads its sections.
python3 -I live-conformance/e2ee_verify.py live-conformance/e2ee

5
.github/wai-conformance/verifiers/energy-steering.sh vendored
View file

@ -2,6 +2,7 @@
# order: 190
# features: energy_steering
# paths: wai/extensions/energy-steering.md
# independent: energy-steering live python3 -I live-conformance/energy_steering_verify.py live-conformance/steering
#
# wai.live.steering (extensions/energy-steering.md): the reference module's unit tests
# (policies and decisions round-trip and their rules refuse, the window, the request
@ -10,7 +11,8 @@
# no figure never placed first by its rank while one with a figure is within the
# limits) and tests/energy_steering.rs (the committed corpus regenerates byte for byte
# from its fixed seeds, wai-rs gives every request and vector its stated outcome, every
# reason of §10 refuses a vector, and §10 lists exactly the reference's codes); then the
# reason of §10 refuses a vector, and §10 lists exactly the reference's codes); and, in
# the step of every independent verifier above, the
# corpus's second implementation, energy_steering_verify.py, written from the
# extension's text and sharing no code with wai-rs, which reads the room with
# live_rooms_verify.py and the per-hop records with live_metering_verify.py, parses RFC
@ -24,4 +26,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib energy_steering::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test energy_steering)
python3 -I live-conformance/energy_steering_verify.py live-conformance/steering

5
.github/wai-conformance/verifiers/exact-splats.sh vendored
View file

@ -2,6 +2,7 @@
# order: 620
# features: splat_exact
# paths: wai/extensions/splat-exact.md wai/extensions/gltf/**
# independent: splat-exact live python3 -I live-conformance/splat_exact_verify.py live-conformance/splat
#
# wai.splat.exact (extensions/splat-exact.md): the reference module's unit tests
# (exp_neg_q16 against the exponential, binary32 ranges rounded half up exactly, the
@ -9,7 +10,8 @@
# frame unchanged) and tests/splat_exact.rs (live-conformance/splat/ regenerates byte for
# byte and no other file is there, every case gets its stated outcome, §10 lists exactly the
# reference's codes, the tiles leave every view of every scene unchanged, and the draft
# schema names §3.2's members); then the corpus's second implementation,
# schema names §3.2's members); and, in the step of every independent verifier above, the
# corpus's second implementation,
# splat_exact_verify.py, written from the extension's text and sharing no code with wai-rs:
# it reads each glTF asset, decodes each payload with int_payload_verify.py's parser and
# rANS decoder and tools/wai_ops_check.py's whm1/1 interpreter, renders every view with its
@ -27,4 +29,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib splat_exact::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test splat_exact)
python3 -I live-conformance/splat_exact_verify.py live-conformance/splat

5
.github/wai-conformance/verifiers/exact-tonemap.sh vendored
View file

@ -2,6 +2,7 @@
# order: 610
# features: exact_tonemap
# paths: wai/extensions/exact-tonemap.md
# independent: tonemap live python3 -I live-conformance/tonemap_verify.py live-conformance/tonemap
#
# wai.media.tonemap, wai.meta.gainmap and wai.meta.st2094_50 (extensions/exact-tonemap.md):
# the reference module's unit tests (the fixed-point log2 and exp2 at known points, the
@ -10,7 +11,8 @@
# (the committed corpus regenerates byte for byte, no other file is there, every case and
# envelope gets its stated outcome, §9 lists exactly the reference's codes, §6 states the
# corpus's measurement against binary64, and frames.rgb8 is the int_video clip's decode
# whose digests its sidecar publishes); then the corpus's second implementation,
# whose digests its sidecar publishes); and, in the step of every independent verifier
# above, the corpus's second implementation,
# tonemap_verify.py, written from the extension's text and sharing no code with wai-rs,
# which computes both derivations in integer arithmetic of its own, reads PNG, envelopes
# (with staged_verify.py) and both companions itself, recomputes §6's measurement, verifies
@ -27,4 +29,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib -- exact_tonemap:: png_still::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test exact_tonemap)
python3 -I live-conformance/tonemap_verify.py live-conformance/tonemap

5
.github/wai-conformance/verifiers/exact-upscale.sh vendored
View file

@ -2,6 +2,7 @@
# order: 310
# features: exact_upscale neural_int
# paths: wai/extensions/exact-upscale.md
# independent: upscale live python3 -I live-conformance/upscale_verify.py live-conformance/upscale
#
# wai.media.upscale (extensions/exact-upscale.md): the integer super-resolution
# derivations' unit tests (the wai-ops/1 graph against the arithmetic of §3.2 written out
@ -12,7 +13,8 @@
# byte from its fixed seeds and its two pinned trained priors, and no other file is there;
# wai-rs gives every case its stated outcome and statements; §8 lists exactly the
# reference's codes, each of which the corpus gives; sha256.json names every accepted
# output); then the corpus's second implementation, upscale_verify.py, written from the
# output); and, in the step of every independent verifier above, the corpus's second
# implementation, upscale_verify.py, written from the
# extension's text and sharing no code with wai-rs, which runs the derivations with
# _int_sr.py, verifies the claims with synthesis_verify.py and the residuals with
# staged_verify.py, and requires every mutated copy of the corpus to fail; standard
@ -27,4 +29,3 @@ set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib exact_upscale::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test exact_upscale)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test cli_derive)
python3 -I live-conformance/upscale_verify.py live-conformance/upscale

5
.github/wai-conformance/verifiers/exploration.sh vendored
View file

@ -2,6 +2,7 @@
# order: 560
# features: exploration
# paths: wai/extensions/exploration.md
# independent: exploration live python3 -I live-conformance/exploration_verify.py live-conformance/exploration
#
# wai.media.exploration (extensions/exploration.md): the reference module's unit tests (the
# descriptors of flat, single-row and checkerboard pictures, grid bins at both ends, the
@ -10,7 +11,8 @@
# regenerates byte for byte from its fixed seeds and no other file is there, wai-rs gives
# every generation, descriptor vector and case its stated outcome, §13 lists exactly the
# reference's codes, each of which the corpus gives, and §3.2 names the reference's
# descriptors); then the corpus's second implementation, exploration_verify.py, written from
# descriptors); and, in the step of every independent verifier above, the corpus's second
# implementation, exploration_verify.py, written from
# the extension's text and sharing no code with wai-rs, which makes every picture with the
# generator as the extension's Appendix A restates it, computes the descriptors, replays
# every archive's schedule, reads usage statements with ai_usage_verify.py, verifies every
@ -26,4 +28,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib exploration::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test exploration)
python3 -I live-conformance/exploration_verify.py live-conformance/exploration

5
.github/wai-conformance/verifiers/fleet.sh vendored
View file

@ -2,6 +2,7 @@
# order: 440
# features: fleet
# paths: wai/extensions/fleet.md
# independent: fleet live python3 -I live-conformance/fleet_verify.py live-conformance/fleet
#
# wai.delivery.fleet (extensions/fleet.md): the reference module's unit tests (the enrolment's
# rules, a summary line split only in the relay's form, a row summed across classes told from
@ -10,7 +11,8 @@
# vector its stated outcome; every reason of §10 and every reused code refuses a vector; the
# accepted vectors state measured, unmetered and partial energy as §5.4 does; the week's
# records held as a service holds them sign the corpus's report byte for byte; §10 lists
# exactly the reference's codes); then the corpus's second implementation, fleet_verify.py,
# exactly the reference's codes); and, in the step of every independent verifier above, the
# corpus's second implementation, fleet_verify.py,
# written from the extension's text and sharing no code with wai-rs, which reads per-hop
# records with live_metering_verify.py, checks every signature with WAI's strict Ed25519
# rule, recomputes every line, row, unmetered count and citation, requires every mutated
@ -25,4 +27,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib fleet::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test fleet)
python3 -I live-conformance/fleet_verify.py live-conformance/fleet

5
.github/wai-conformance/verifiers/forensic-watermark.sh vendored
View file

@ -2,6 +2,7 @@
# order: 670
# features: live_watermark
# paths: wai/extensions/forensic-watermark.md
# independent: live-watermark live python3 -I live-conformance/live_watermark_verify.py live-conformance/watermark
#
# wai.live.watermark (extensions/forensic-watermark.md): the reference module's unit tests
# (the CBOR reader's deterministic-encoding rules, tokens of both COSE structures and
@ -9,7 +10,8 @@
# the exact match and the binding) and tests/live_watermark.rs (vectors.json regenerates
# byte for byte from its fixed seeds and no other file is there, wai-rs gives every
# token, session list, match and disclosure its stated outcome, and §11 lists exactly the
# reference's codes, each of which the corpus gives); then the corpus's second
# reference's codes, each of which the corpus gives); and, in the step of every
# independent verifier above, the corpus's second
# implementation, live_watermark_verify.py, written from the extension's text and sharing
# no code with wai-rs, which reads rooms with live_rooms_verify.py and tokens with a CBOR
# reader of its own, requires every mutated copy of the corpus to fail, and drops each of
@ -23,4 +25,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib live_watermark::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test live_watermark)
python3 -I live-conformance/live_watermark_verify.py live-conformance/watermark

5
.github/wai-conformance/verifiers/holdings.sh vendored
View file

@ -2,12 +2,14 @@
# order: 550
# features: holdings
# paths: wai/extensions/holdings.md
# independent: holdings live needs=cryptography python3 -I live-conformance/holdings_verify.py live-conformance/holdings
#
# wai.delivery.holdings (extensions/holdings.md): the reference module's unit tests (an entry's
# one form, a record read back and every rule refusing, queries and a plan, and the corpus
# reaching every stated outcome and failing a changed one) and tests/holdings.rs (vectors.json
# regenerates byte for byte from its fixed seeds, wai-rs gives every case its stated outcome,
# and §7 lists exactly the reference's codes); then the corpus's second implementation,
# and §7 lists exactly the reference's codes); and, in the step of every independent verifier
# above, the corpus's second implementation,
# holdings_verify.py, written from the extension's text and sharing no code with wai-rs, which
# verifies group receipts with jwp_verify.py and signatures with the shared _ed25519.py, writes
# the record, its verdict, queries and plans itself, reads §7, requires every mutated copy of
@ -19,4 +21,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib -- holdings::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test holdings)
python3 -I live-conformance/holdings_verify.py live-conformance/holdings

2
.github/wai-conformance/verifiers/http.sh vendored
View file

@ -1,5 +1,6 @@
# step: http_verify.py
# order: 50
# independent: http delivery needs=cryptography python3 http-conformance/http_verify.py http-conformance
#
# The HTTP delivery corpus's second implementation (http-conformance/README): the head,
# the head proof (every committed proof rebuilt from the text), response checks, plans,
@ -12,4 +13,3 @@
# `cryptography` package installed above (without it those cases fail, never skip).
# Its mutations and the differential against wai-rs run in wai-http.yml's falsify
# job.
python3 http-conformance/http_verify.py http-conformance

5
.github/wai-conformance/verifiers/hybrid-signatures.sh vendored
View file

@ -3,13 +3,15 @@
# build: pq
# features: pq_signatures synthesis,pq_signatures
# paths: wai/extensions/pq-signatures.md
# independent: pq trust needs=cryptography python3 -I pq-conformance/pq_verify.py pq-conformance
#
# Hybrid signatures (extensions/pq-signatures.md): the reference module's unit tests (its
# ML-DSA-65 against FIPS 204's ACVP vectors — keyGen, deterministic sigGen and sigVer —
# vendored in pq-conformance/acvp and pinned by SHA-256, and the composite against the
# draft's own vector, before anything is built on them), the synthesis reader's hybrid
# records, and tests/pq_signatures.rs (vectors.json regenerates byte for byte, every case
# reaches its verdict, §8 lists exactly the reference's codes, each given); then the
# reaches its verdict, §8 lists exactly the reference's codes, each given); and, in the
# step of every independent verifier above, the
# corpus's second implementation, pq_verify.py, written from the extension's text and
# sharing no code with wai-rs, which verifies ML-DSA-65 from FIPS 204's text in pure
# Python (_mldsa.py), checks every ACVP verification vector and signature and the draft's
@ -25,7 +27,6 @@ set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_PQ_FEATURES" --lib pq_sig::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_PQ_FEATURES" --lib synthesis::pq_tests)
(cd wai-rs && cargo test --no-default-features --features "$WAI_PQ_FEATURES" --test pq_signatures)
python3 -I pq-conformance/pq_verify.py pq-conformance
python3 -I pq-conformance/pq_verify.py verdicts pq-conformance/vectors.json > "$RUNNER_TEMP/pq-py.tsv"
(cd wai-rs && "$RUNNER_TEMP/wai-bins/pq/wai_pq_vectors" verdicts ../pq-conformance/vectors.json) > "$RUNNER_TEMP/pq-rs.tsv"
diff "$RUNNER_TEMP/pq-py.tsv" "$RUNNER_TEMP/pq-rs.tsv"

5
.github/wai-conformance/verifiers/influence.sh vendored
View file

@ -2,6 +2,7 @@
# order: 490
# features: influence
# paths: wai/extensions/influence.md
# independent: influence live python3 -I live-conformance/influence_verify.py live-conformance/influence
#
# wai.media.influence (extensions/influence.md): the reference module's unit tests (the
# blend keeps its base and its references by weight, the makers' split is whole, an
@ -9,7 +10,8 @@
# committed corpus regenerates byte for byte from its fixed seeds and no other file is
# there, wai-rs gives every case, statement and report its stated outcome, every reason of
# §9 and every code it reuses refuses a vector, and §9 lists exactly the reference's codes);
# then the corpus's second implementation, influence_verify.py, written from the
# and, in the step of every independent verifier above, the corpus's second implementation,
# influence_verify.py, written from the
# extension's text and sharing no code with wai-rs, which verifies each session with
# contribution_verify.py, regenerates every conditioned step with that implementation's
# arithmetic of the generator, reads usage statements with ai_usage_verify.py and
@ -25,4 +27,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib influence::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test influence)
python3 -I live-conformance/influence_verify.py live-conformance/influence

2
.github/wai-conformance/verifiers/int-payload.sh vendored
View file

@ -1,9 +1,9 @@
# step: int_payload_verify.py
# order: 10
# independent: int-payload trust python3 int-payload-conformance/int_payload_verify.py int-payload-conformance
#
# The integer payload corpus's second implementation (int-payload-conformance/README):
# it parses every payload with the extension's rules and check order, converts every
# JSON source independently and requires the committed bytes, decodes the WIV1
# keyframe-0 payloads in full to their sidecars' SHA-256, and checks every refusal
# fixture. Standard library only, so it needs nothing installed.
python3 int-payload-conformance/int_payload_verify.py int-payload-conformance

5
.github/wai-conformance/verifiers/key-release.sh vendored
View file

@ -2,6 +2,7 @@
# order: 240
# features: key_release
# paths: wai/extensions/key-release.md
# independent: key-release live python3 -I live-conformance/key_release_verify.py live-conformance/keys
#
# The proxy form of wai.video.keys (extensions/key-release.md): the reference module's unit
# tests (the commitment list, the entitlement and the ticket round-trip and their rules
@ -10,7 +11,8 @@
# from a CPIX document), form 1's unit tests (its legacy pin unchanged), and
# tests/key_release.rs (every file of the committed corpus regenerates byte for byte from
# its fixed seeds and no other file is there, every log gets its stated outcome, and the
# extension's tables are the reference's codes); then the corpus's second implementation,
# extension's tables are the reference's codes); and, in the step of every independent
# verifier above, the corpus's second implementation,
# key_release_verify.py, written from the extension's text and sharing no code with wai-rs,
# which reads the extension's tables, derives every list from its CPIX document with the
# standard library's XML parser, recomputes every decision of every log, verifies every
@ -25,4 +27,3 @@ set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib key_release::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib wai_video_keys::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test key_release)
python3 -I live-conformance/key_release_verify.py live-conformance/keys

5
.github/wai-conformance/verifiers/live-ads.sh vendored
View file

@ -2,6 +2,7 @@
# order: 230
# features: live_ads
# paths: wai/extensions/live-ads.md
# independent: live-ads live python3 -I live-conformance/live_ads_verify.py live-conformance/ads
#
# wai.live.ads (extensions/live-ads.md): the reference module's unit tests (policies and
# annotations round-trip and their rules refuse, a break's identity, tags, events and
@ -10,7 +11,8 @@
# unchanged), and tests/live_ads.rs (every file of the committed corpus, vectors.json and
# each room's HLS tags, regenerates byte for byte from its fixed seeds and no other file
# is there, wai-rs gives every room its stated outcome, and §10 lists exactly the
# reference's codes, each of which the corpus gives); then the corpus's second
# reference's codes, each of which the corpus gives); and, in the step of every
# independent verifier above, the corpus's second
# implementation, live_ads_verify.py, written from the extension's text and sharing no
# code with wai-rs, which verifies each room with live_rooms_verify.py, writes every tag,
# event and asset list byte for byte, verifies every receipt with WAI's strict Ed25519
@ -26,4 +28,3 @@ set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib live_ads::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib wai_video_ssai::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test live_ads)
python3 -I live-conformance/live_ads_verify.py live-conformance/ads

5
.github/wai-conformance/verifiers/live-compose.sh vendored
View file

@ -2,13 +2,15 @@
# order: 130
# features: live_compose
# paths: wai/extensions/live-compose.md
# independent: live-compose live python3 -I live-conformance/live_compose_verify.py live-conformance/compose
#
# wai.live.compose (extensions/live-compose.md): the reference module's unit tests (the
# layout over every grid of 1 to 64 tiles stays on its canvas with no tile overlapping
# another, the fill modes, the attributes, the mixer) and tests/live_compose.rs (the
# committed corpus regenerates byte for byte from its fixed seeds, wai-rs gives every
# case its stated outcome, every reason of §10 refuses a case, and §10 lists exactly
# the reference's codes); then the corpus's second implementation,
# the reference's codes); and, in the step of every independent verifier above, the
# corpus's second implementation,
# live_compose_verify.py, written from the extension's text and sharing no code with
# wai-rs, which verifies each room with live_rooms_verify.py, steps through its
# lifecycle replay to each instant itself, gives every layout, mix, descriptor,
@ -22,4 +24,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib live_compose::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test live_compose)
python3 -I live-conformance/live_compose_verify.py live-conformance/compose

3
.github/wai-conformance/verifiers/live-contribution.sh vendored
View file

@ -1,6 +1,7 @@
# step: live contribution · live_contribution_verify.py
# order: 380
# paths: wai/extensions/live-contribution.md
# independent: live-contribution live python3 -I live-conformance/live_contribution_verify.py live-conformance/links
#
# wai.live.contribution (extensions/live-contribution.md): the link records' corpus
# under live-conformance/links/, which joule-moq-ingest's reference generates and its
@ -15,5 +16,3 @@
# The paths this step brings to the workflow's trigger, and why:
# wai/extensions/live-contribution.md: live_contribution_verify.py reads this extension's §7 list
# of the link record's reasons and requires the corpus to raise each one.
set -euo pipefail
python3 -I live-conformance/live_contribution_verify.py live-conformance/links

5
.github/wai-conformance/verifiers/live-events.sh vendored
View file

@ -2,12 +2,14 @@
# order: 100
# features: live_events
# paths: wai/extensions/live-events.md
# independent: live-events live python3 -I live-conformance/live_events_verify.py live-conformance/events
#
# wai.live.events (extensions/live-events.md): the reference module's unit tests and
# tests/live_events.rs (the committed corpus regenerates byte for byte from its fixed
# seeds, wai-rs gives every vector, policy case and binding case its stated outcome,
# the corpus covers every kind and reason, and §9 lists exactly the reference's
# codes); then the corpus's second implementation, live_events_verify.py, written
# codes); and, in the step of every independent verifier above, the corpus's second
# implementation, live_events_verify.py, written
# from the extension's text and sharing no code with wai-rs, which verifies the same
# cases with WAI's strict Ed25519 (claim-conformance/_ed25519.py) and BLAKE3
# (claim-conformance/_blake3.py), standard library only, run isolated (-I).
@ -18,4 +20,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib live_events::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test live_events)
python3 -I live-conformance/live_events_verify.py live-conformance/events

5
.github/wai-conformance/verifiers/live-messaging.sh vendored
View file

@ -2,11 +2,13 @@
# order: 660
# features: live_messaging
# paths: wai/extensions/live-messaging.md
# independent: live-messaging live python3 -I live-conformance/live_messaging_verify.py live-conformance/messaging
#
# wai.live.message and wai.live.call (extensions/live-messaging.md): tests/live_messaging.rs
# (vectors.json regenerates byte for byte from its fixed seeds and no other file is there,
# wai-rs gives every conversation, history request, call and record its stated outcome, and
# §12 lists exactly the reference's codes, each of which the corpus gives); then the corpus's
# §12 lists exactly the reference's codes, each of which the corpus gives); and, in the step
# of every independent verifier above, the corpus's
# second implementation, live_messaging_verify.py, written from the extension's text and
# sharing no code with wai-rs, which reads descriptors with live_rooms_verify.py and Key IDs,
# keys, AES-GCM and plaintexts with e2ee_verify.py, implements the receipts' binary form and
@ -21,4 +23,3 @@
# wire identifiers test reads its sections.
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test live_messaging)
python3 -I live-conformance/live_messaging_verify.py live-conformance/messaging

9
.github/wai-conformance/verifiers/live-metering.sh vendored
View file

@ -2,6 +2,7 @@
# order: 180
# features: live_metering
# paths: wai/extensions/live-metering.md
# independent: live-metering live python3 -I live-conformance/live_metering_verify.py live-conformance/metering
#
# wai.live.metering (extensions/live-metering.md): the reference module's unit tests
# (records and ledgers round-trip, the energy rules, the size bound at its limit, a
@ -10,9 +11,10 @@
# tests/live_metering.rs (the committed corpus regenerates byte for byte from its fixed
# seeds and the reference relay's recorded record, wai-rs gives every vector its stated
# outcome, every reason of §9 refuses a vector, the large audience's relay record is the
# lesson's size, and §9 lists exactly the reference's codes); then the corpus's second
# implementation, live_metering_verify.py, written from the extension's text and sharing
# no code with wai-rs, which verifies each room with live_rooms_verify.py, each take with
# lesson's size, and §9 lists exactly the reference's codes); and, in the step of every
# independent verifier above, the corpus's second implementation,
# live_metering_verify.py, written from the extension's text and sharing no code with
# wai-rs, which verifies each room with live_rooms_verify.py, each take with
# capture_take_verify.py and each composition receipt with live_compose_verify.py,
# recomputes every unit, total and grant's consumption itself, and requires every
# mutated copy of the corpus to fail; standard library only, run isolated (-I).
@ -24,4 +26,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib live_metering::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test live_metering)
python3 -I live-conformance/live_metering_verify.py live-conformance/metering

5
.github/wai-conformance/verifiers/live-moderation.sh vendored
View file

@ -2,6 +2,7 @@
# order: 210
# features: live_moderation
# paths: wai/extensions/live-moderation.md
# independent: live-moderation live python3 -I live-conformance/live_moderation_verify.py live-conformance/moderation
#
# wai.live.moderation (extensions/live-moderation.md): the reference module's unit tests
# (policies, statements and notices round-trip and their rules refuse, the export's
@ -9,7 +10,8 @@
# tests/live_moderation.rs (every file of the committed corpus, vectors.json and each
# statement-of-reasons export, regenerates byte for byte from its fixed seeds and no
# other file is there, wai-rs gives every room, export and clip its stated outcome, and
# §10 lists exactly the reference's codes, each of which the corpus gives); then the
# §10 lists exactly the reference's codes, each of which the corpus gives); and, in the
# step of every independent verifier above, the
# corpus's second implementation, live_moderation_verify.py, written from the
# extension's text and sharing no code with wai-rs, which verifies each room with
# live_rooms_verify.py and each clip with live_record_verify.py, writes every export
@ -24,4 +26,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib live_moderation::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test live_moderation)
python3 -I live-conformance/live_moderation_verify.py live-conformance/moderation

5
.github/wai-conformance/verifiers/live-provenance.sh vendored
View file

@ -2,6 +2,7 @@
# order: 280
# features: live_provenance
# paths: wai/extensions/live-provenance.md
# independent: live-provenance live python3 -I live-conformance/c2pa_live_verify.py live-conformance/c2pa-live
#
# wai.live.provenance (extensions/live-provenance.md): the reference module's unit tests
# (boxes, chunks, emsg and the BMFF hash; the JUMBF store; COSE_Sign1 and the OKP key;
@ -10,7 +11,8 @@
# committed corpus regenerates byte for byte from its fixed seeds and the packaging
# corpus's AV1 floor, and no other file is there; wai-rs gives every vector its stated
# outcome; every reason of §12 refuses a vector; §12 lists exactly the reference's codes
# with the two C2PA codes it reports); then the corpus's second implementation,
# with the two C2PA codes it reports); and, in the step of every independent verifier
# above, the corpus's second implementation,
# c2pa_live_verify.py, written from the extension's text and sharing no code with
# wai-rs, which reads CBOR, ISO BMFF, JUMBF, COSE and X.509 DER itself, verifies every
# signature with WAI's strict Ed25519 rule and every receipt with BLAKE3 from
@ -24,4 +26,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib live_provenance::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test live_provenance)
python3 -I live-conformance/c2pa_live_verify.py live-conformance/c2pa-live

5
.github/wai-conformance/verifiers/live-recording.sh vendored
View file

@ -2,13 +2,15 @@
# order: 140
# features: live_record
# paths: wai/extensions/live-recording.md
# independent: live-recording live python3 -I live-conformance/live_record_verify.py live-conformance/recordings
#
# wai.live.recording (extensions/live-recording.md): the reference module's unit tests,
# the output tree's (whose range proofs it adds, tested against RFC 9162 §2.1.3.2's
# check), and tests/live_record.rs (the committed corpus regenerates byte for byte from
# its fixed seeds, wai-rs gives every recording and clip its stated outcome, every reason
# of §9 appears in it, every take it says it reuses is capture binding's byte for byte,
# and §9 lists exactly the reference's codes); then the corpus's second implementation,
# and §9 lists exactly the reference's codes); and, in the step of every independent
# verifier above, the corpus's second implementation,
# live_record_verify.py, written from the extension's text and sharing no code with
# wai-rs, which checks range proofs by the extension's recursion and by RFC 9162
# §2.1.3.2's own algorithm, verifies each take with capture_take_verify.py and every
@ -21,4 +23,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib -- live_record:: output_tree::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test live_record)
python3 -I live-conformance/live_record_verify.py live-conformance/recordings

12
.github/wai-conformance/verifiers/live-rooms.sh vendored
View file

@ -2,15 +2,16 @@
# order: 110
# features: live_room
# paths: wai/extensions/live-rooms.md
# independent: live-rooms live python3 -I live-conformance/live_rooms_verify.py live-conformance/rooms
#
# wai.live.room (extensions/live-rooms.md): the reference module's unit tests and
# tests/live_room.rs (the committed corpus regenerates byte for byte from its fixed
# seeds, wai-rs gives every vector its stated outcome, every reason of §11 refuses a
# vector, and §11 lists exactly the reference's codes); then the corpus's second
# implementation, live_rooms_verify.py, written from the extension's text and sharing
# no code with wai-rs, which verifies every event with live_events_verify.py and every
# signature with WAI's strict Ed25519 (claim-conformance/_ed25519.py), standard
# library only, run isolated (-I).
# vector, and §11 lists exactly the reference's codes); and, in the step of every
# independent verifier above, the corpus's second implementation, live_rooms_verify.py,
# written from the extension's text and sharing no code with wai-rs, which verifies
# every event with live_events_verify.py and every signature with WAI's strict Ed25519
# (claim-conformance/_ed25519.py), standard library only, run isolated (-I).
#
# The paths this step brings to the workflow's trigger, and why:
# wai/extensions/live-rooms.md: tests/live_room.rs requires this extension's §11 to list the codes
@ -18,4 +19,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib live_room::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test live_room)
python3 -I live-conformance/live_rooms_verify.py live-conformance/rooms

5
.github/wai-conformance/verifiers/live-worlds.sh vendored
View file

@ -2,12 +2,14 @@
# order: 640
# features: live_world
# paths: wai/extensions/live-worlds.md
# independent: live-worlds live python3 -I live-conformance/live_worlds_verify.py live-conformance/worlds
#
# wai.live.world (extensions/live-worlds.md): the reference module's unit tests (the
# statement and its rules, the grant attribute) and tests/live_world.rs (vectors.json
# regenerates byte for byte from its fixed seeds and no other file is there, wai-rs gives
# every world its stated outcome, and §11 lists exactly the reference's codes, each of which
# a verifier raises the corpus gives); then the corpus's second implementation,
# a verifier raises the corpus gives); and, in the step of every independent verifier above,
# the corpus's second implementation,
# live_worlds_verify.py, written from the extension's text and the sections it cites and
# sharing no code with wai-rs: the statement, each grant's signature and attributes, each
# receipt in its binary form under the owner's chain, each op's signature, owner, kind,
@ -19,4 +21,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib live_world::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test live_world)
python3 -I live-conformance/live_worlds_verify.py live-conformance/worlds

2
.github/wai-conformance/verifiers/loc.sh vendored
View file

@ -1,5 +1,6 @@
# step: loc_verify.py
# order: 760
# independent: loc live python3 -I live-conformance/loc_verify.py live-conformance/loc
#
# LOC carriage (extensions/loc-carriage.md): the second implementation of joule-moq's loc
# module (whose tests/loc_vectors.rs runs in joule-code-conformance), written from the text,
@ -7,4 +8,3 @@
# every object back to its envelope, writes it again byte for byte, gives every refused
# object and envelope its code, and requires every mutated copy, and the corpus under a
# copy of itself with a check dropped, to fail.
python3 -I live-conformance/loc_verify.py live-conformance/loc

5
.github/wai-conformance/verifiers/media-identification.sh vendored
View file

@ -3,10 +3,12 @@
# features: identify
# alone: -
# paths: wai/extensions/media-identify.md
# independent: identify delivery python3 -I identify-conformance/identify_verify.py identify-conformance
#
# Media identification (extensions/media-identify.md): wai-rs's unit tests (hostile bytes
# through every walk, what a file reads as its bytes) and tests/identify.rs (the corpus
# regenerates byte for byte with every result; the extension's tables; `wai identify`); then
# regenerates byte for byte with every result; the extension's tables; `wai identify`); and,
# in the step of every independent verifier above,
# identify_verify.py, written from the text and sharing no code with wai-rs but
# format_probe_verify.py's probes, which gives every file its whole result, holds ext/ to
# what an independent prober read of it, and requires every mutated copy to fail.
@ -19,4 +21,3 @@ set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib identify::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test identify)
(cd wai-rs && "$RUNNER_TEMP/wai-bins/union/wai_identify_vectors" verify ../identify-conformance)
python3 -I identify-conformance/identify_verify.py identify-conformance

5
.github/wai-conformance/verifiers/native-sink.sh vendored
View file

@ -2,6 +2,7 @@
# order: 460
# features: native_sink
# paths: wai/extensions/native-sink.md
# independent: native-sink live needs=cryptography python3 -I live-conformance/native_sink_verify.py live-conformance/native-sink
#
# The native sink (extensions/native-sink.md): the reference module's unit tests (dates,
# durations and attribute lists read as a player reads them, origins serialized as RFC 6454
@ -10,7 +11,8 @@
# one refused for another owner or channel) and tests/native_sink.rs (vectors.json
# regenerates byte for byte from its seeds and no other file is there, every call gets the
# answer it states, §8 lists exactly the reasons the corpus reaches, and every case run
# through the C interface's symbols gets the same answers); then the corpus's second
# through the C interface's symbols gets the same answers); and, in the step of every
# independent verifier above, the corpus's second
# implementation, native_sink_verify.py, written from the extension's text and sharing no
# code with wai-rs, which replays every case, reads playlist receipts and their chain with
# http_verify.py, holds every bundle to its own session field by field and verifies it with
@ -24,4 +26,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib native_sink::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test native_sink)
python3 -I live-conformance/native_sink_verify.py live-conformance/native-sink

11
.github/wai-conformance/verifiers/oer.sh vendored Normal file
View file

@ -0,0 +1,11 @@
# step: oer2_verify.py
# order: 780
# independent: oer trust needs=cryptography python3 oer-conformance/oer2_verify.py oer-conformance/vectors.json
# independent: oer-firmware trust needs=cryptography python3 oer-conformance/oer2_verify.py oer-conformance/firmware_vectors.json
# independent: oer-edge trust needs=cryptography python3 oer-conformance/oer2_verify.py oer-conformance/edge_vectors.json
# independent: oer-job trust needs=cryptography python3 oer-conformance/oer2_verify.py oer-conformance/job_vectors.json
# independent: oer-unreadable trust python3 oer-conformance/oer2_verify.py --unreadable oer-conformance/unreadable
#
# The OER corpus's second implementation (oer-conformance/README): oer2_verify.py over each
# of the corpus's four vectors files, each a run and a record of its own, and over its
# unreadable documents (--unreadable). No workflow ran it before the union.

5
.github/wai-conformance/verifiers/opportunity-report.sh vendored
View file

@ -2,6 +2,7 @@
# order: 420
# features: opportunity_report
# paths: wai/extensions/opportunity-report.md
# independent: opportunity-report live needs=cryptography python3 -I live-conformance/opportunity_report_verify.py live-conformance/opportunity
#
# wai.delivery.opportunity (extensions/opportunity-report.md): the reference module's unit
# tests (the SCTE 35 subset reader against the standard's sample and damaged sections, the
@ -11,7 +12,8 @@
# outcome; every reason of §13 and every reused code refuses a vector; §13 lists exactly the
# reference's codes; the worked example, a channel's day whose reported statement
# under-reports one owner's programme, and the check that shows it, recomputed from the
# public records alone); then the corpus's second implementation, opportunity_report_verify.py,
# public records alone); and, in the step of every independent verifier above, the corpus's
# second implementation, opportunity_report_verify.py,
# written from the extension's text and sharing no code with wai-rs, which verifies rooms
# with live_ads_verify.py and contracts, runs and signed statements with
# revenue_share_verify.py, reads every report, check, section and reported statement itself,
@ -27,4 +29,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib opportunity_report::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test opportunity_report)
python3 -I live-conformance/opportunity_report_verify.py live-conformance/opportunity

8
.github/wai-conformance/verifiers/pin.sh vendored Normal file
View file

@ -0,0 +1,8 @@
# step: pin_verify.py
# order: 770
# independent: pin trust python3 pin-conformance/pin_verify.py pin-conformance
#
# The pin corpus's second implementation: Python standard library only, sharing no
# code with wai-rs. It checks digests and listing rules, which do not depend on the
# architecture, so one run is enough: it ran on cargo-classes' macOS leg, and runs in the
# union now.

5
.github/wai-conformance/verifiers/platform-encode.sh vendored
View file

@ -2,13 +2,15 @@
# order: 570
# features: platform_encode
# paths: wai/extensions/platform-encode.md
# independent: encode delivery python3 -I encode-conformance/encode_verify.py encode-conformance
#
# wai.video.encode (extensions/platform-encode.md): the reference module's unit tests (the
# Exp-Golomb reader, emulation prevention, NAL splitting, length-prefixed samples, the
# resampler, the source reader, the strict JSON scan and the date-time) and
# tests/platform_encode.rs (the fixture streams are the pinned bytes, every generated file
# regenerates byte for byte, every case gets its outcome, every record reason has a case, and
# §12 lists exactly the reference's reasons); then the corpus's second implementation,
# §12 lists exactly the reference's reasons); and, in the step of every independent verifier
# above, the corpus's second implementation,
# encode_verify.py, written from the extension's text and sharing no code with wai-rs or
# wai-pack, which reads every Annex B stream, its access units and parameter sets itself,
# verifies every record with WAI's strict Ed25519 rule and every object with BLAKE3 from
@ -23,4 +25,3 @@
set -euo pipefail
(cd wai-rs && cargo test --locked --no-default-features --features "$WAI_FEATURES" --lib platform_encode::)
(cd wai-rs && cargo test --locked --no-default-features --features "$WAI_FEATURES" --test platform_encode)
python3 -I encode-conformance/encode_verify.py encode-conformance

5
.github/wai-conformance/verifiers/prior-forms.sh vendored
View file

@ -2,6 +2,7 @@
# order: 330
# features: prior_wire prior_wire,neural_int
# paths: wai/extensions/prior-carriage.md
# independent: prior-forms live python3 -I live-conformance/prior_forms_verify.py live-conformance/prior-forms
#
# Tensor files as prior forms (extensions/prior-carriage.md §9 to §14): the module's unit
# tests (each layout read strictly and written canonically, the arithmetic coder's integers
@ -9,7 +10,8 @@
# bindings through every form) and tests/prior_wire.rs (the committed corpus regenerates
# byte for byte from its pinned native artifacts; §13 lists exactly the reference's codes;
# every corpus prior in every form loads to its native bytes, and the two trained WSR1
# priors give the same upscaled picture as their native files); then the corpus's second
# priors give the same upscaled picture as their native files); and, in the step of every
# independent verifier above, the corpus's second
# implementation, prior_forms_verify.py, written from the extension's text and sharing no
# code with wai-rs, which also requires every byte-flipped copy of an accepted file to be
# caught; standard library only, run isolated (-I).
@ -21,4 +23,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib -- prior_wire:: strict_json:: codecs::prior_form)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test prior_wire)
python3 -I live-conformance/prior_forms_verify.py live-conformance/prior-forms

5
.github/wai-conformance/verifiers/qoe-aggregation.sh vendored
View file

@ -2,6 +2,7 @@
# order: 600
# features: telemetry_aggregate
# paths: wai/extensions/telemetry-aggregation.md
# independent: telemetry-aggregate live python3 -I live-conformance/telemetry_aggregate_verify.py live-conformance/aggregate
#
# wai.telemetry.aggregate (extensions/telemetry-aggregation.md): the reference module's unit
# tests (the fields against a wide-arithmetic-free product, the generators' orders,
@ -11,7 +12,8 @@
# no other file is there, vdaf-22/ is the draft's test vectors as published and wai-rs passes
# every one, every receipt, cell and delivery-only report gets its stated outcome, every
# reason of §11 and every reused code refuses a vector, §11 lists exactly the reference's
# codes, delivery-telemetry §3.4 classes exactly the reference's keys); then the corpus's
# codes, delivery-telemetry §3.4 classes exactly the reference's keys); and, in the step of
# every independent verifier above, the corpus's
# second implementation, telemetry_aggregate_verify.py, written from the extension's text and
# the drafts it cites and sharing no code with wai-rs, which evaluates Prio3 itself and passes
# the draft's vectors, opens every report with its own X25519, HKDF, AES-128-GCM and HPKE,
@ -29,4 +31,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib -- telemetry_aggregate:: telemetry::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test telemetry_aggregate)
python3 -I live-conformance/telemetry_aggregate_verify.py live-conformance/aggregate

3
.github/wai-conformance/verifiers/quantum-referee.sh vendored
View file

@ -1,6 +1,6 @@
# step: quantum referee.py
# order: 700
# working-directory: .
# independent: quantum media python3 -I quantum-conformance/tools/referee.py quantum-conformance
#
# The quantum corpus's second implementation (quantum-sim extension), integers only:
# it derives the circuit/1 phase table by its recurrence and the circuit2 table by an
@ -12,4 +12,3 @@
# to see every ±1 change of every P(k) under both tables. Standard library only, run
# isolated (-I), so no module from the working directory, the environment or the
# user's site-packages is imported.
python3 -I wai/quantum-conformance/tools/referee.py wai/quantum-conformance

8
.github/wai-conformance/verifiers/receipt-ledger.sh vendored
View file

@ -2,6 +2,7 @@
# order: 400
# features: receipt_ledger
# paths: wai/extensions/receipt-ledger.md
# independent: receipt-ledger live needs=cryptography python3 -I live-conformance/receipt_ledger_verify.py live-conformance/ledger
#
# wai.delivery.ledger (extensions/receipt-ledger.md): the reference module's unit tests and
# tests/receipt_ledger.rs (vectors.json regenerates byte for byte from its fixed seeds and
@ -10,9 +11,9 @@
# stated outcome; every reason of §10 and every reused code refuses a vector, and so does
# each source kind's own extension; §10 lists exactly the reference's codes; under a tariff
# of version 3, jobs priced per operation and a chain that bills a job's work once, its
# output's receipt cited again by a re-run, with version 2 billing as before); then the
# corpus's second implementation, receipt_ledger_verify.py, written from the extension's
# text and sharing no code with wai-rs, which verifies each source with that corpus's own
# output's receipt cited again by a re-run, with version 2 billing as before); and, in the step of every
# independent verifier above, the corpus's second implementation, receipt_ledger_verify.py, written from the
# extension's text and sharing no code with wai-rs, which verifies each source with that corpus's own
# second implementation (live_metering_verify.py, settlement_verify.py, live_ads_verify.py
# with ad_presentation_verify.py, stitched_presentation_verify.py, pack_verify.py, and for a
# media job route_record_verify.py and _int_motion_receipt.py) and
@ -31,4 +32,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib receipt_ledger::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test receipt_ledger)
python3 -I live-conformance/receipt_ledger_verify.py live-conformance/ledger

5
.github/wai-conformance/verifiers/remote-desktop.sh vendored
View file

@ -2,6 +2,7 @@
# order: 590
# features: remote_desktop
# paths: wai/extensions/remote-desktop.md
# independent: remote-desktop live python3 -I live-conformance/remote_desktop_verify.py live-conformance/remote
#
# wai.live.remote (extensions/remote-desktop.md): the reference module's unit tests (every
# statement read back and refused when a byte is short or over, the screen statement's
@ -13,7 +14,8 @@
# wai-rs gives every room its stated outcome at the commitment level and at the opened
# level, and §12 lists exactly the reference's codes, host refusals, floor refusals and
# the reasons a seal is not opened, each of which the corpus gives);
# then the corpus's second implementation, remote_desktop_verify.py, written from the
# and, in the step of every independent verifier above, the corpus's second implementation,
# remote_desktop_verify.py, written from the
# extension's text and sharing no code with wai-rs, which verifies each room with
# live_rooms_verify.py, every signature with WAI's strict Ed25519 rule and every seal with
# the aggregation verifier's HPKE, recomputes every outcome from the rules, the floor and
@ -31,5 +33,4 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib -- remote_desktop:: hpke::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test remote_desktop)
python3 -I live-conformance/remote_desktop_verify.py live-conformance/remote
node --test ../joule-code/joule-code-rs/crates/joule-moq-web/web/room/remote.test.mjs

5
.github/wai-conformance/verifiers/revenue-share.sh vendored
View file

@ -2,6 +2,7 @@
# order: 410
# features: revenue_share
# paths: wai/extensions/revenue-share.md
# independent: revenue-share live needs=cryptography python3 -I live-conformance/revenue_share_verify.py live-conformance/share
#
# wai.delivery.revenue_share (extensions/revenue-share.md): the reference module's unit tests
# (the largest-remainder division and its ties, the date-time a marker states, the contract's
@ -10,7 +11,8 @@
# seeds and the ledger corpus, and no other file is there; wai-rs gives every vector its
# stated outcome; every reason of §11 and every reused code refuses a vector; §11 lists exactly
# the reference's codes; the worked example, three statements of one month computed and then
# recomputed from the public records alone; the four operations from the command line); then
# recomputed from the public records alone; the four operations from the command line); and, in
# the step of every independent verifier above,
# the corpus's second implementation, revenue_share_verify.py, written from the extension's
# text and sharing no code with wai-rs, which verifies each ledger document with
# receipt_ledger_verify.py, reads every contract, run, marker and statement itself,
@ -27,4 +29,3 @@ set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib revenue_share::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib wai_video_channel::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test revenue_share)
python3 -I live-conformance/revenue_share_verify.py live-conformance/share

5
.github/wai-conformance/verifiers/robustness.sh vendored
View file

@ -3,6 +3,7 @@
# features: robustness
# release-features: robustness
# paths: wai/extensions/robustness-evidence.md
# independent: robustness live python3 -I live-conformance/robustness_verify.py live-conformance/robustness
#
# Robustness evidence for soft bindings (extensions/robustness-evidence.md): the
# reference's unit tests (gamma is the rounded power, rates are exact and reduced, every
@ -13,7 +14,8 @@
# is there; every case gets what it states; §12 lists exactly the reference's codes and
# the corpus refuses a case for each; §4 registers exactly the reference's transforms;
# the tent64 records' tables are the synthesis corpus's battery.json; the tent64 claim is
# synthesis §7.5.8's list; jwp-receipts §3.1 lists the span reasons); then the second
# synthesis §7.5.8's list; jwp-receipts §3.1 lists the span reasons); and, in the step of
# every independent verifier above, the second
# implementation, robustness_verify.py with _wai_mark.py, written from the extension's
# text and sharing no code with wai-rs, which reruns the conformance record's battery end
# to end and requires the same table, checks the tent64 records against battery.json and
@ -27,4 +29,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib robustness::)
(cd wai-rs && cargo test --release --no-default-features --features "$WAI_RELEASE_FEATURES" --test robustness)
python3 -I live-conformance/robustness_verify.py live-conformance/robustness

5
.github/wai-conformance/verifiers/route-record.sh vendored
View file

@ -2,6 +2,7 @@
# order: 450
# features: route_record
# paths: wai/extensions/route-record.md
# independent: route-record live needs=cryptography python3 -I live-conformance/route_record_verify.py live-conformance/routes
#
# wai.route.record (extensions/route-record.md): the reference module's unit tests (a
# document read strictly and held to the subset, §2's encodings, an estimate past the subset
@ -11,7 +12,8 @@
# file is there; wai-rs gives every vector its stated outcome; every reason of §8 and every
# reused code refuses a vector; the accepted vectors are §10's, of version 1 and of version 2,
# every limit removing a candidate; a record is bound to each of its documents; `wai route
# verify` prints the outcome and exits by it); then the corpus's second implementation,
# verify` prints the outcome and exits by it); and, in the step of every independent verifier
# above, the corpus's second implementation,
# route_record_verify.py, written from the extension's text and sharing no code with wai-rs,
# which checks the router's signature with WAI's strict Ed25519 rule and the hop receipts
# with the OER/2 reference verifier (the `cryptography` package, installed above), computes
@ -27,4 +29,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib route_record::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test route_record)
python3 -I live-conformance/route_record_verify.py live-conformance/routes

5
.github/wai-conformance/verifiers/scene-carriage.sh vendored
View file

@ -2,13 +2,15 @@
# order: 340
# features: scene_map
# paths: wai/extensions/scene-carriage.md
# independent: scene live python3 -I live-conformance/scene_verify.py live-conformance/scene
#
# Scene carriage (extensions/scene-carriage.md): the module's unit tests (the conversions
# and spellings of §3 against fixed values, the containers an import builds equal to the
# ones wai-rs's world and audio-scene modules write, base64 and the USD subset's reader)
# and tests/scene_map.rs (the committed corpus regenerates byte for byte: every bundle
# written as glTF and USD and read back, every refusal reached; §8 lists exactly the
# reference's codes); then the corpus's second implementation, scene_verify.py, written
# reference's codes); and, in the step of every independent verifier above, the corpus's
# second implementation, scene_verify.py, written
# from the extension's text and sharing no code with wai-rs, which also requires every
# byte-flipped copy of an exported document to be caught; standard library only, run
# isolated (-I).
@ -20,4 +22,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib -- scene_map:: strict_json::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test scene_map)
python3 -I live-conformance/scene_verify.py live-conformance/scene

5
.github/wai-conformance/verifiers/settlement.sh vendored
View file

@ -2,6 +2,7 @@
# order: 390
# features: settlement
# paths: wai/extensions/settlement.md
# independent: settlement live python3 -I live-conformance/settlement_verify.py live-conformance/settlement
#
# wai.delivery.settlement (extensions/settlement.md): the reference module's unit tests
# (range proofs over every range around every chunk boundary of objects of many lengths
@ -10,7 +11,8 @@
# tests/settlement.rs (vectors.json regenerates byte for byte from its fixed seeds and no
# other file is there, wai-rs gives every statement and range proof its stated outcome, §11
# lists exactly the reference's codes, each of which the corpus gives, beside every code it
# reuses); then the corpus's second implementation, settlement_verify.py, written from the
# reuses); and, in the step of every independent verifier above, the corpus's second
# implementation, settlement_verify.py, written from the
# extension's text and sharing no code with wai-rs, which verifies range proofs over
# BLAKE3's chunk tree with claim-conformance's BLAKE3, reads rooms, grants and proofs with
# live_rooms_verify.py and per-hop records with live_metering_verify.py, checks signatures
@ -26,4 +28,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib settlement::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test settlement)
python3 -I live-conformance/settlement_verify.py live-conformance/settlement

5
.github/wai-conformance/verifiers/sink-energy.sh vendored
View file

@ -2,6 +2,7 @@
# order: 200
# features: sink_energy
# paths: wai/extensions/sink-energy.md
# independent: sink-energy live python3 -I live-conformance/sink_energy_verify.py live-conformance/sink-energy
#
# wai.sink.energy (extensions/sink-energy.md): the reference module's unit tests (a
# bracket summary is its intervals in integers, a round's figure and least declaration
@ -13,7 +14,8 @@
# the wrap; on macOS, the per-process counter reads this process) and
# tests/sink_energy.rs (the committed corpus regenerates byte for byte from its fixed
# seeds, every case gets its stated outcome, every reason of §8 refuses a case, and every
# measured table under tables/ is accepted with its readings and procedure); then the
# measured table under tables/ is accepted with its readings and procedure); and, in the
# step of every independent verifier above, the
# corpus's second implementation, sink_energy_verify.py, written from the extension's text
# and sharing no code with wai-rs, which recomputes every round, row, segment and split in
# integers and requires every mutated copy of the corpus to fail; standard library only,
@ -28,4 +30,3 @@ set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib sink_meter::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test sink_energy)
(cd wai-rs && "$RUNNER_TEMP/wai-bins/union/wai_sink_energy_vectors" verify ../live-conformance/sink-energy)
python3 -I live-conformance/sink_energy_verify.py live-conformance/sink-energy

2
.github/wai-conformance/verifiers/staged-measure.sh vendored
View file

@ -1,9 +1,9 @@
# step: measure_verify.py
# order: 70
# independent: staged-measure delivery python3 staged-measure-conformance/measure_verify.py staged-measure-conformance
#
# The staged-measurement corpus's second implementation (staged-measure-conformance/README):
# it counts every object, applies every layer from its base's stage 0, computes the quality
# integers, the output tree, the prefixes, the overhead points and the dedup totals itself,
# and requires its decode to be the fixture tool's and its figures to be goldens.json's.
# Standard library only.
python3 staged-measure-conformance/measure_verify.py staged-measure-conformance

2
.github/wai-conformance/verifiers/staged.sh vendored
View file

@ -1,9 +1,9 @@
# step: staged_verify.py
# order: 40
# independent: staged delivery python3 staged-conformance/staged_verify.py staged-conformance
#
# The staged-delivery corpus's second implementation (staged-conformance/README): it
# replays every case from its stage-0 buffers through its own reading of the
# extension (refines, WIR1, the sink's steps, holds, batches, conflicts, deadlines,
# tiers, stage hashes), decodes the WIV1 keyframe-0, PNG and FLAC bases, explodes the
# WAI2 files, and replays the schedule. Standard library only.
python3 staged-conformance/staged_verify.py staged-conformance

5
.github/wai-conformance/verifiers/stitched-presentation.sh vendored
View file

@ -2,6 +2,7 @@
# order: 260
# features: stitched_presentation
# paths: wai/extensions/stitched-presentation.md
# independent: stitched-presentation live python3 -I live-conformance/stitched_presentation_verify.py live-conformance/stitched
#
# wai.video.stitched_presentation (extensions/stitched-presentation.md): the reference
# module's unit tests (claims and pod-input records round-trip and their rules refuse,
@ -11,7 +12,8 @@
# regenerates byte for byte from its fixed seeds and no other file is there, wai-rs gives
# every record, claim and ad its stated facts and every vector its spend by currency, and
# §9 lists exactly the codes the reference adds); then the tool's own check of the corpus
# through its binary; then the corpus's second implementation,
# through its binary; and, in the step of every independent verifier above, the corpus's
# second implementation,
# stitched_presentation_verify.py, written from the extension's text and sharing no code
# with wai-rs, which reads every playlist, asset list, MPD, VAST document, form-1 receipt,
# record and claim itself, requires every mutated copy of the corpus to fail and catches
@ -29,5 +31,4 @@ set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib live_compose::layout::frame_tests)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test stitched_presentation)
(cd wai-rs && "$RUNNER_TEMP/wai-bins/union/wai_stitched_presentation" vectors verify ../live-conformance/stitched)
python3 -I live-conformance/stitched_presentation_verify.py live-conformance/stitched
node --test ../joule-code/joule-code-rs/crates/joule-moq-web/web/ads/stitched.test.mjs

5
.github/wai-conformance/verifiers/stream-audit.sh vendored
View file

@ -2,13 +2,15 @@
# order: 430
# features: stream_audit
# paths: wai/extensions/stream-audit.md
# independent: stream-audit live python3 -I live-conformance/stream_audit_verify.py live-conformance/audit
#
# wai.delivery.audit (extensions/stream-audit.md): the reference module's unit tests (the
# acknowledgement windows, chains and repeats of §4.2, the origin rule, the report's bytes,
# channel names) and tests/stream_audit.rs (vectors.json regenerates byte for byte from its
# fixed seeds and no other file is there, wai-rs gives every session and report its stated
# outcome, §14 lists exactly the reference's codes, each of which the corpus gives, beside
# every code it borrows); then the corpus's second implementation, stream_audit_verify.py,
# every code it borrows); and, in the step of every independent verifier above, the corpus's
# second implementation, stream_audit_verify.py,
# written from the extension's text and sharing no code with wai-rs, which reads
# acknowledgements with settlement_verify.py, claims with ad_presentation_verify.py and logs
# and energy classes with live_metering_verify.py, checks every signature with WAI's strict
@ -24,4 +26,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib stream_audit::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test stream_audit)
python3 -I live-conformance/stream_audit_verify.py live-conformance/audit

5
.github/wai-conformance/verifiers/stream-termination.sh vendored
View file

@ -2,6 +2,7 @@
# order: 630
# features: stream_termination
# paths: wai/extensions/stream-termination.md
# independent: stream-termination live python3 -I live-conformance/stream_termination_verify.py live-conformance/termination
#
# wai.live.termination (extensions/stream-termination.md) and the display match of
# capability-constraints §4: the reference module's unit tests (sides without orientation,
@ -10,7 +11,8 @@
# regenerates byte for byte from its fixed seeds, wai-rs gives every match and log its
# stated outcome and facts, every reason refuses a log, §7 lists exactly the reference's
# codes and capability-constraints §4.2 its match bits); wai-live-web's functions for a
# page held to the same corpus natively, a page's log written through them verified; then
# page held to the same corpus natively, a page's log written through them verified; and,
# in the step of every independent verifier above,
# the corpus's second implementation, stream_termination_verify.py, written from the two
# extensions' text and sharing no code with wai-rs, which recomputes every match and every
# log's outcome and facts and requires every mutated copy of the corpus to fail; standard
@ -27,4 +29,3 @@ set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test stream_termination)
(cd wai-rs && "$RUNNER_TEMP/wai-bins/union/wai_stream_termination_vectors" verify ../live-conformance/termination)
(cd wai-live-web && cargo test --locked --features int_video --lib termination_tests)
python3 -I live-conformance/stream_termination_verify.py live-conformance/termination

5
.github/wai-conformance/verifiers/synthesis-mark.sh vendored
View file

@ -3,6 +3,7 @@
# features: synthesis_mark
# release-features: synthesis_mark
# paths: wai/extensions/synthesis.md
# independent: synthesis-mark live python3 -I live-conformance/synthesis_mark_verify.py live-conformance/synthesis-mark
#
# The registered watermark schemes wai-mark:tent64 and wai-mark:tent64s
# (extensions/synthesis.md §7.4 to §7.9): the reference's unit tests (the code corrects 29
@ -15,7 +16,8 @@
# every embedding, detection and case gets what it states; the extension's definitions,
# generator polynomial and shuffles are the reference's; §7.5.8 and §7.9.7 claim only
# attacks the measurements show survived, and §7.5.6's and §7.9.5's figures are the
# measurements'); then the second implementation, synthesis_mark_verify.py with
# measurements'); and, in the step of every independent verifier above, the second
# implementation, synthesis_mark_verify.py with
# _wai_mark.py, written from the extension's text and sharing no code with wai-rs, which
# embeds, detects and decodes itself, both stages of the tiled detector among them, runs
# the battery's picture attacks, derives the frames the vectors name, verifies every case
@ -31,4 +33,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib -- synthesis::mark synthesis_mark::)
(cd wai-rs && cargo test --release --no-default-features --features "$WAI_RELEASE_FEATURES" --test synthesis_mark)
python3 -I live-conformance/synthesis_mark_verify.py live-conformance/synthesis-mark

5
.github/wai-conformance/verifiers/synthesis.sh vendored
View file

@ -2,13 +2,15 @@
# order: 290
# features: synthesis
# paths: wai/extensions/synthesis.md
# independent: synthesis live python3 -I live-conformance/synthesis_verify.py live-conformance/synthesis
#
# wai.media.synthesis (extensions/synthesis.md): the reference module's unit tests (the
# pinned derivations by their rules, the class lattice, a generator's envelope) and
# tests/synthesis.rs (the committed corpus regenerates byte for byte from its fixed seeds
# and no other file is there, wai-rs gives every case its stated outcome and statements,
# §12 lists exactly the reference's codes, each of which the corpus gives, and §10.2's
# table pins exactly the reference's icons); then the corpus's second implementation,
# table pins exactly the reference's icons); and, in the step of every independent
# verifier above, the corpus's second implementation,
# synthesis_verify.py, written from the extension's text and sharing no code with wai-rs,
# which runs the pinned derivations and writes each digital source type and C2PA
# statement itself, verifies each room with live_rooms_verify.py and every record with
@ -25,4 +27,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib synthesis::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test synthesis)
python3 -I live-conformance/synthesis_verify.py live-conformance/synthesis

5
.github/wai-conformance/verifiers/takedown.sh vendored
View file

@ -2,6 +2,7 @@
# order: 480
# features: takedown
# paths: wai/extensions/takedown.md
# independent: takedown live python3 -I live-conformance/takedown_verify.py live-conformance/takedown
#
# wai.media.takedown (extensions/takedown.md): the reference module's unit tests (times,
# clock identifiers, the published registry, the order of subjects, the lower median) and
@ -9,7 +10,8 @@
# and no other file is there, wai-rs gives every case its stated outcome and facts and
# every counts entry its rows, §11 lists exactly the reference's codes, each of which the
# corpus gives, and the registry the module carries is registry/takedown-clocks.json);
# then the corpus's second implementation, takedown_verify.py, written from the
# and, in the step of every independent verifier above, the corpus's second
# implementation, takedown_verify.py, written from the
# extension's text and sharing no code with wai-rs, which computes every closure, holder,
# purge time, clock verdict and counts row itself, reads derivation records and delivery
# receipts with edge_verify.py, origin claims with synthesis_verify.py, acknowledgements
@ -27,4 +29,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib takedown::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test takedown)
python3 -I live-conformance/takedown_verify.py live-conformance/takedown

5
.github/wai-conformance/verifiers/timed-text.sh vendored
View file

@ -2,6 +2,7 @@
# order: 150
# features: timed_text
# paths: wai/extensions/timed-text.md
# independent: timed-text live python3 -I live-conformance/timed_text_verify.py live-conformance/captions
#
# wai.text.timed (extensions/timed-text.md): the reference module's unit tests (cue model,
# annotation, WebVTT, IMSC1 and its render-model bound, timelines, disclosure, and the
@ -9,7 +10,8 @@
# vectors.json and each expected .vtt, .ttml and .608.hex, regenerates byte for byte
# from its fixed seeds and no other file is there, wai-rs gives every case its stated
# outcome, the corpus covers every 608 code path, and §12 lists exactly the reference's
# codes); then the corpus's second implementation, timed_text_verify.py, written from
# codes); and, in the step of every independent verifier above, the corpus's second
# implementation, timed_text_verify.py, written from
# the extension's text and sharing no code with wai-rs, which verifies every event with
# live_events_verify.py, writes every output byte for byte from the text, parses each
# IMSC1 document with xml.etree, and requires every mutated copy of the corpus to fail;
@ -21,4 +23,3 @@
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib timed_text::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test timed_text)
python3 -I live-conformance/timed_text_verify.py live-conformance/captions

10
.github/workflows/repo-checks.yml vendored
View file

@ -100,6 +100,16 @@ jobs:
- name: "workflow hygiene: every wasmtime is the pinned release, run with both determinism settings"
if: ${{ !cancelled() }}
run: python3 -I scripts/check-wasmtime-pin.py
# ── The release runs the quickstart from each archive it attaches ───────
# wai-release.yml runs only on a tag, so a change that stops it running the
# quickstart, or lets it fail, would first be seen at a release. This reads the
# workflow and the two scripts on every change, for the cost of a few lines of text.
- name: "release quickstart: the check fails closed (self-test)"
if: ${{ !cancelled() }}
run: bash scripts/check-wai-release-quickstart-selftest.sh
- name: "release quickstart: each archive runs it before it is attached, and the twin keeps its steps"
if: ${{ !cancelled() }}
run: python3 -I scripts/check-wai-release-quickstart.py
# ── Code cites no section its SPEC lacks; WAI's cited sections exist ────
- name: "spec references: the check can fail (self-test)"

16
.github/workflows/wai-breaking.yml vendored
View file

@ -13,11 +13,14 @@ name: WAI breaking changes and the conformance runner
# changelog entry names it) and of `wai conform` (the reference implementations conform, a
# changed verdict or output byte fails, a changed record is refused by `wai verify`, and
# `wai verify --conformant` admits a passing run of the native sink and refuses a planted
# failing one); then `wai conform` runs the corpora's independent Python verifiers and the
# reference packager, and `wai verify` checks every record those runs wrote; and it runs the
# native sink, through its module and its C interface, and the score decoder, each record
# admitted against this checkout's corpus and a planted failing sink refused
# (conformance-results §4.1).
# failing one; the JUnit and HTML reports are the records'; `--independent` holds the
# corpus's independent verifier to the same cases and fails on a disagreement, and alone
# runs the verifiers it names); then `wai conform` runs the corpora's independent Python
# verifiers and the reference packager, and `wai verify` checks every record those runs
# wrote; and it runs the native sink, through its module and its C interface, and the score
# decoder, each record admitted against this checkout's corpus and a planted failing sink
# refused (conformance-results §4.1). The corpus union itself, every independent verifier
# through `wai conform --independent`, runs in wai-conformance.yml.
#
# Push filters match the push's delta, and the `relevant` job gates the work from the pull
# request's own diff (scripts/pr-touches-paths.py, reading the list below), as wai-pack.yml
@ -36,6 +39,9 @@ on:
# The toolchain pin decides which compiler documents both sides of the Rust API.
- 'rust-toolchain.toml'
- 'scripts/pr-touches-paths.py'
# wai-rs/build.rs reads `wai conform`'s table of independent verifiers from the step files
# of wai-conformance.yml, and the runner's tests below hold the table to them.
- '.github/wai-conformance/verifiers/**'
- '.github/workflows/wai-breaking.yml'
jobs:

57
.github/workflows/wai-conformance.yml vendored
View file

@ -438,12 +438,16 @@ jobs:
# outside its roots; WAI Studio at /_wai/, its API to this origin alone) and the project
# file's reader. tests/cli_studio.rs: `wai dev` against a stand-in relay and room service
# serves Studio at /_wai/, every socket the stack listens on is on loopback, and the
# local wai runs an operation for the page.
- name: The floor build, and wai hello, create, dev and verify
# local wai runs an operation for the page. tests/cli_doctor.rs: `wai doctor` with PATH,
# cargo's home and the install laid out by the test: this wai first holds, another
# program called wai first fails and is named (the crates.io package wai-cli's as that),
# and an install with nothing for `wai dev` to run fails; on each leg's own system, for
# the ports, browsers and keychain it reads.
- name: The floor build, and wai hello, create, dev, verify and doctor
working-directory: wai/wai-rs
run: |
cargo test --no-default-features --features floor --test floor
cargo test --no-default-features --features quickstart --bin wai --test floor --test cli_dx --test cli_fleet --test cli_studio
cargo test --no-default-features --features quickstart --bin wai --test floor --test cli_dx --test cli_fleet --test cli_studio --test cli_doctor
# tests/cli_http.rs runs `wai http publish`, `codecs`, `origin` and `bench` as a user
# does: objects named by their content hash with their head proofs, a sequence's union
# and floor, the origin's headers, ranges and If-Range on loopback, and a fetch by
@ -586,14 +590,6 @@ jobs:
- name: Every cargo-class corpus reproduces its frozen hashes
working-directory: wai/wai-rs
run: python3 ../../scripts/wai-conformance-lists.py corpora --verify --extra-features provenance,video_manifest
# The pin corpus's second implementation: Python standard library only, sharing no
# code with wai-rs. One leg is enough — it checks digests and listing rules, which
# do not depend on the architecture — and the macOS host has python3.
- name: The pin corpus checks against an independent verifier (macOS)
if: matrix.os == 'macos'
shell: bash
working-directory: wai
run: python3 pin-conformance/pin_verify.py pin-conformance
# Every committed audio vector must come from a seeded generator. This check reads
# each one to PCM (each audio base's stage 0 in the staged corpus, each measured
# source over one, each spatial-audio source, every audio file under wai/) and
@ -1038,8 +1034,45 @@ jobs:
python3 ../../scripts/wai-conformance-lists.py prebuild > "$RUNNER_TEMP/wai-prebuild.sh"
cat "$RUNNER_TEMP/wai-prebuild.sh"
bash --noprofile --norc -eo pipefail "$RUNNER_TEMP/wai-prebuild.sh"
# The corpus union, one command: every corpus's independent verifier, its second
# implementation, written from the extension's text and sharing no code with wai-rs, run
# by `wai conform --independent` in wai/ with the command line its README gives. The list
# is the table `wai conform` holds (`wai conform --list` prints it), which build.rs reads
# from the `# independent:` lines of the step files below, so a corpus's verifier joins it
# in its own step's file; the table's tests require every *_verify.py of the tree to be in
# it or to say where it runs instead, and no step to run one of them by hand. What each
# verifier checks is in its corpus's README and its own header; those of the claim, JWP,
# key-discovery, HTTP, ledger, route and post-quantum corpora, and the ones that read
# them, check Ed25519's equation with the `cryptography` package installed above (without
# it their cases fail, never skip). Each run is a signed result record, which `wai verify`
# checks; the JUnit report goes with the records, and a failed verifier's output is in the
# log. `wai` is built with the feature quickstart into a target directory of its own, so
# the union's builds above stay as they are. The steps below are each corpus's Rust side.
- name: Every corpus's independent verifier, through wai conform --independent
working-directory: wai
shell: bash
run: |
set -euo pipefail
(cd wai-rs && cargo build --locked --no-default-features --features quickstart --bin wai --target-dir "$RUNNER_TEMP/wai-cli-target")
w="$RUNNER_TEMP/wai-cli-target/debug/wai"
d="$RUNNER_TEMP/independent"
mkdir -p "$d"
status=0
"$w" conform --independent --root . --out "$d/records" --format junit > "$d/junit.xml" || status=$?
"$w" verify "$d/records"
exit "$status"
# An upload this forge cannot take must not decide the job: the verdict is the step above.
- name: Upload the independent verifiers' records and report
if: always()
continue-on-error: true
uses: actions/upload-artifact@v3
with:
name: independent-verifiers
path: ${{ runner.temp }}/independent
# Every step, in its order, each in a group of the log; the first that fails stops the
# job, as a failed step did, and is named in an error line with its file.
# job, as a failed step did, and is named in an error line with its file. A file that
# declares verifiers and runs nothing itself (no body) is no step: the union ran them.
- name: Every verifier step (.github/wai-conformance/verifiers/)
shell: bash
run: python3 scripts/wai-conformance-lists.py run

63
.github/workflows/wai-release.yml vendored
View file

@ -12,10 +12,17 @@ name: wai command line release
# The first job checks the tag against the crate's version and the installer's, and that each
# project file schema (wai/schema/) is served at its $id as committed: every project file the
# release writes names its schema there, so a tag does not build until the site serves it.
# Each archive's own `wai` runs before it is attached (create --list, hello, verify, and on
# the hosts that run it, dev --check). A last job writes the Homebrew formula and the Scoop
# manifest for the archives attached and attaches them too; they go into the tap and the bucket
# by hand.
# Before it is attached, each archive runs the quickstart as a new user runs it, from the
# archive alone on a fresh home directory and a PATH of the system's own directories
# (wai/quickstart/test/clean-machine.sh; on Windows its PowerShell twin, clean-machine.ps1):
# the archive checked against its SHA-256 and unpacked, what it holds checked (every binary,
# each of which runs, the room client and WAI Studio), then wai hello, wai verify,
# wai create my-room --template video-room and wai dev --check in the project, Studio at /_wai/
# among what it checks; and on the Apple-silicon Mac, the project's page joined in two browser
# pages, each decoding the other. A cross-built archive, whose binaries this host cannot run,
# is not run.
# A last job writes the Homebrew formula and the Scoop manifest for the archives attached and
# attaches them too; they go into the tap and the bucket by hand.
#
# Uploads go to the forge directly over the private network (FORGE_API), never through the
# public tunnel. A runner that is down leaves its platform's archive missing: the release lists
@ -115,9 +122,12 @@ jobs:
strategy:
fail-fast: false
matrix:
# `browser`: whether the host has the browser and the automation package the
# quickstart's two-page step drives (the Apple-silicon runner does, as wai-examples.yml
# uses them; the Intel Mac is not set up for it).
include:
- { triple: aarch64-apple-darwin, runner: macos }
- { triple: x86_64-apple-darwin, runner: macos-x86_64 }
- { triple: aarch64-apple-darwin, runner: macos, browser: true }
- { triple: x86_64-apple-darwin, runner: macos-x86_64, browser: false }
steps:
- uses: actions/checkout@v5
- name: Put cargo on PATH
@ -132,11 +142,27 @@ jobs:
curl -fsS -o "$RUNNER_TEMP/web.tar.gz" "${FORGE_API%/api/v1}/$FORGE_REPO/releases/download/$GITHUB_REF_NAME/wai-web-$v.tar.gz"
tar -C "$RUNNER_TEMP" -xzf "$RUNNER_TEMP/web.tar.gz"
nice -n 10 bash wai/quickstart/build.sh "$RUNNER_TEMP/dist" --web "$RUNNER_TEMP/web" --studio "$RUNNER_TEMP/studio" --target "${{ matrix.triple }}" --version "$v"
if [ "${{ matrix.triple }}" = "$(rustc -vV | sed -n 's/^host: //p')" ]; then
t="$RUNNER_TEMP/try"; mkdir -p "$t"; tar -C "$t" -xzf "$RUNNER_TEMP/dist/wai-$v-${{ matrix.triple }}.tar.gz"
b="$t/wai-$v-${{ matrix.triple }}/bin"
(cd "$t" && "$b/wai" hello h >/dev/null && "$b/wai" verify h >/dev/null && mkdir p && cd p && "$b/wai" dev --check --port 0)
- name: The quickstart from the archive, as a new user runs it
shell: bash
env:
BROWSER_BIN: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome # allow-name: the macOS runner's browser executable, a file path
run: |
set -euo pipefail
v="${{ needs.version.outputs.version }}"
if [ "${{ matrix.triple }}" != "$(rustc -vV | sed -n 's/^host: //p')" ]; then
echo "cross-built for ${{ matrix.triple }}: this host cannot run it"
exit 0
fi
flag=""
if [ "${{ matrix.browser }}" = true ]; then
test -x "$BROWSER_BIN" || { echo "::error::this runner has no browser at $BROWSER_BIN"; exit 1; }
pw="$RUNNER_TEMP/automation"; mkdir -p "$pw"
cp wai/quickstart/test/package.json wai/quickstart/test/package-lock.json "$pw/"
(cd "$pw" && npm ci --ignore-scripts --no-audit --no-fund > /dev/null)
export PLAYWRIGHT_HOST="$pw/node_modules"
flag=--browser
fi
bash wai/quickstart/test/clean-machine.sh "$RUNNER_TEMP/dist/wai-$v-${{ matrix.triple }}.tar.gz" $flag
- name: Attach to the release
shell: bash
env:
@ -184,9 +210,9 @@ jobs:
curl -fsS -o "$RUNNER_TEMP/web.tar.gz" "${FORGE_API%/api/v1}/$FORGE_REPO/releases/download/$GITHUB_REF_NAME/wai-web-$v.tar.gz"
tar -C "$RUNNER_TEMP" -xzf "$RUNNER_TEMP/web.tar.gz"
bash wai/quickstart/build.sh "$RUNNER_TEMP/dist" --web "$RUNNER_TEMP/web" --studio "$RUNNER_TEMP/studio" --target "$t" --version "$v"
d="$RUNNER_TEMP/try"; mkdir -p "$d"; tar -C "$d" -xzf "$RUNNER_TEMP/dist/wai-$v-$t.tar.gz"
b="$d/wai-$v-$t/bin"
(cd "$d" && "$b/wai" hello h >/dev/null && "$b/wai" verify h >/dev/null && mkdir p && cd p && "$b/wai" dev --check --port 0)
- name: The quickstart from the archive, as a new user runs it
shell: bash
run: bash wai/quickstart/test/clean-machine.sh "$RUNNER_TEMP/dist/wai-${{ needs.version.outputs.version }}-${{ matrix.triple }}.tar.gz"
- name: Attach to the release
shell: bash
env:
@ -244,6 +270,17 @@ jobs:
Compress-Archive -Path $stage -DestinationPath (Join-Path $dist "$name.zip") -Force
$h = (Get-FileHash (Join-Path $dist "$name.zip") -Algorithm SHA256).Hash.ToLower()
Set-Content -NoNewline -Path (Join-Path $dist "$name.zip.sha256") -Value "$h $name.zip`n"
# Not yet run: the Windows runner is down, so this step has never run on a Windows archive.
# Under PowerShell 7 on a macOS archive the script has run without its contents step and
# its check of WAI Studio, which have not run yet.
- name: The quickstart from the archive, as a new user runs it
shell: pwsh
run: |
$ErrorActionPreference = 'Stop'
Remove-Item Env:CI -ErrorAction SilentlyContinue
$name = "wai-${{ needs.version.outputs.version }}-x86_64-pc-windows-msvc"
pwsh -NoProfile -File wai\quickstart\test\clean-machine.ps1 (Join-Path $env:RUNNER_TEMP "dist\$name.zip")
if ($LASTEXITCODE -ne 0) { throw "the quickstart failed from $name.zip" }
- name: Attach to the release
shell: pwsh
env:

36
scripts/check-wai-ops-doors-selftest.sh
View file

@ -3,10 +3,10 @@
# check-wai-ops-doors-selftest.sh — check-wai-ops-doors.py can fail.
#
# A check that cannot fail is decoration. This builds a scratch tree shaped like the
# repository (a catalogue export, an MCP tool table, an OpenAPI document, the C header and a
# Swift and a Kotlin source) whose every door names its operations as the catalogue does, and
# asserts that the check passes it and fails each way a door can misname an operation,
# saying which.
# repository (a catalogue export, the catalogue's source with its verbs, copied from this
# repository, an MCP tool table, an OpenAPI document, the C header and a Swift and a Kotlin
# source) whose every door names its operations as the catalogue does, and asserts that the
# check passes it and fails each way a door can misname an operation, saying which.
set -uo pipefail
here=$(cd "$(dirname "$0")" && pwd)
@ -19,6 +19,7 @@ mk() {
mkdir -p "$tmp/wai/catalogue" "$tmp/wai/wai-rs/src/bin/wai_cli/mcp" "$tmp/wai/wai-rs/include" \
"$tmp/joule-code/joule-code-rs/crates/svc" "$tmp/wai/wai-sink-swift/Sources/WaiSink" \
"$tmp/wai/wai-sink-android/core/src/main/kotlin/wai/sink"
cp "$here/../wai/wai-rs/src/catalogue.rs" "$tmp/wai/wai-rs/src/catalogue.rs"
cat > "$tmp/wai/catalogue/ops.json" <<'J'
{"format": "wai.ops/1", "operations": [
{"id": "envelope.inspect", "placement": "local", "route": null},
@ -164,6 +165,33 @@ expect fail "a Kotlin companion method with a near-synonym verb" "WaiReceipt.des
mk; sub "$tmp/wai/wai-sink-android/core/src/main/kotlin/wai/sink/WaiReceipt.kt" 'private fun checkQuietly' 'fun checkQuietly'
expect fail "a private Kotlin method made public under an operation verb" "WaiReceipt.checkQuietly names receipt.check"
# The verbs are the catalogue's own list, read from its source: each verb it has added since
# naming rule N3 was approved is held, and a near-synonym it lists is refused.
mk; printf 'int32_t wai_envelope_doctor(const uint8_t *p);\n' >> "$tmp/wai/wai-rs/include/wai.h"
expect fail "a C function under the catalogue's verb doctor" "wai_envelope_doctor names envelope.doctor, which the catalogue does not hold"
mk; printf 'int32_t wai_envelope_upscale(const uint8_t *p);\n' >> "$tmp/wai/wai-rs/include/wai.h"
expect fail "a C function under the catalogue's verb upscale" "wai_envelope_upscale names envelope.upscale"
mk; sub "$tmp/wai/wai-sink-swift/Sources/WaiSink/Envelope.swift" 'public func pack() {}' 'public func tonemap() {}'
expect fail "a Swift method under the catalogue's verb tonemap" "WaiEnvelope.tonemap names envelope.tonemap"
mk; printf 'int32_t wai_receipt_up(const uint8_t *p);\n' >> "$tmp/wai/wai-rs/include/wai.h"
expect fail "a C function under the catalogue's near-synonym up" "wai_receipt_up: the verb up is a near-synonym"
mk; sub "$tmp/wai/wai-rs/src/catalogue.rs" '"tonemap", "doctor",' '"tonemap",'
printf 'int32_t wai_envelope_doctor(const uint8_t *p);\n' >> "$tmp/wai/wai-rs/include/wai.h"
expect pass "a verb the catalogue does not list is library API: the list is the catalogue's" "C: 1 functions name operations of the catalogue, each held; 4 are library API"
mk; sub "$tmp/wai/wai-rs/src/catalogue.rs" 'pub const VERBS:' 'pub const RETIRED_VERBS:'
expect fail "a catalogue whose verbs cannot be read" "no operation verbs could be read"
mk; sub "$tmp/wai/wai-rs/src/catalogue.rs" 'pub const NEAR_SYNONYMS:' 'pub const RETIRED_NEAR:'
expect fail "a catalogue whose near-synonyms cannot be read" "no near-synonyms could be read"
mk; rm "$tmp/wai/wai-rs/src/catalogue.rs"
expect fail "no catalogue source to read the verbs from" "catalogue.rs"
if [ "$fails" -ne 0 ]; then
echo "check-wai-ops-doors-selftest: $fails case(s) failed"
exit 1

47
scripts/check-wai-ops-doors.py
View file

@ -19,10 +19,13 @@ What each door is held to:
- HTTP API: every operationId is an operation of the catalogue that runs on a service
(placement `service` or `both`), at the route the catalogue gives it.
- C, Swift, Kotlin: a name in a catalogue noun's namespace whose verb is an operation verb
(the closed list below) names an operation, and that operation must be in the catalogue;
a near-synonym verb (`get`, `describe`, `info`, `keygen`, `init`, `new`, `up`) in a catalogue
(naming rule N3's closed list) names an operation, and that operation must be in the
catalogue; a near-synonym verb (`get` for `show`, `init` for `create`, …) in a catalogue
noun's namespace is refused. Any other name is library API the catalogue does not list
(the C ABI's codec and sink functions, the native sinks' classes), and is reported as such.
The verbs and their near-synonyms are the catalogue's own lists, `VERBS` and
`NEAR_SYNONYMS` in wai/wai-rs/src/catalogue.rs, read from it: there is one list, and a verb
the catalogue gains is held here from the change that adds it.
A door that has no operation of the catalogue yet is reported as such; that is not a
failure. A wrong name is.
@ -37,19 +40,12 @@ import os
import re
import sys
# The operation verbs: the closed list of WAI toolchain v1 (naming rule N3), with the two it
# gained after its approval, upscale and tonemap (wai/catalogue/README.md). The catalogue's
# own copy is wai/wai-rs/src/catalogue.rs's VERBS; a test there holds this one to it.
VERBS = {
"create", "list", "show", "update", "delete", "check", "verify", "run", "build", "export",
"import", "start", "stop", "join", "leave", "grant", "invite", "revoke", "rotate", "tail",
"search", "test", "load", "encode", "decode", "inspect", "wrap", "unwrap", "fetch",
"generate", "promote", "rollback", "lock", "upgrade", "setup", "guide", "stream",
"upscale", "tonemap",
}
# Near-synonyms of an operation verb, and the verb to use: the catalogue's NEAR_SYNONYMS
# (wai/wai-rs/src/catalogue.rs), which a test there holds this one to.
NEAR = {"get": "show", "describe": "show", "info": "show", "keygen": "create", "init": "create", "new": "create", "up": "start"}
# The catalogue's source: its closed list of operation verbs (naming rule N3) and their
# near-synonyms, each with the verb to use.
CATALOGUE = "wai/wai-rs/src/catalogue.rs"
# Read from CATALOGUE by main().
VERBS = set()
NEAR = {}
MCP_TABLE = "wai/wai-rs/src/bin/wai_cli/mcp/tools.rs"
OPENAPI_GLOBS = ["joule-code/joule-code-rs/crates/*/openapi.json", "wai/*/openapi.json", "wai/wai-rs/openapi.json"]
@ -66,6 +62,23 @@ def load_ops(path):
return {o["id"]: o for o in doc["operations"]}
def load_verbs(path):
"""(verbs, near-synonyms) from the catalogue's source: `pub const VERBS: &[&str] = &[…];`
and `pub const NEAR_SYNONYMS: &[(&str, &str)] = &[(…), …];`. A source either list cannot be
read from stops the check: a door held to no verb would hold nothing."""
with open(path, encoding="utf-8") as f:
src = f.read()
m = re.search(r"pub const VERBS: &\[&str\] = &\[(.*?)\];", src, re.S)
verbs = set(re.findall(r'"([a-z0-9_]+)"', m.group(1))) if m else set()
if not verbs:
raise SystemExit(f"{path}: no operation verbs could be read (`pub const VERBS: &[&str] = &[...];`)")
m = re.search(r"pub const NEAR_SYNONYMS: &\[\(&str, &str\)\] = &\[(.*?)\];", src, re.S)
near = dict(re.findall(r'\(\s*"([^"]+)"\s*,\s*"([^"]+)"\s*\)', m.group(1))) if m else {}
if not near:
raise SystemExit(f"{path}: no near-synonyms could be read (`pub const NEAR_SYNONYMS: &[(&str, &str)] = &[...];`)")
return verbs, near
def split_id(i):
n, _, v = i.partition(".")
return n, v
@ -277,6 +290,8 @@ def main(argv):
print("usage: check-wai-ops-doors.py [--root DIR] [--ops FILE]", file=sys.stderr)
return 2
ops = load_ops(ops_path or os.path.join(root, "wai/catalogue/ops.json"))
global VERBS, NEAR
VERBS, NEAR = load_verbs(os.path.join(root, CATALOGUE))
nouns = {split_id(i)[0] for i in ops}
findings, notes = [], []
check_mcp(root, ops, findings, notes)
@ -288,7 +303,7 @@ def main(argv):
print(f"ok {n}")
for f in findings:
print(f"FAIL {f}")
print(f"check-wai-ops-doors: {len(ops)} operations; {len(findings)} finding(s)")
print(f"check-wai-ops-doors: {len(ops)} operations, {len(VERBS)} verbs; {len(findings)} finding(s)")
return 1 if findings else 0

151
scripts/check-wai-release-quickstart-selftest.sh Executable file
View file

@ -0,0 +1,151 @@
#!/usr/bin/env bash
#
# check-wai-release-quickstart-selftest.sh — check-wai-release-quickstart.py can fail.
#
# A check that cannot fail is decoration. This copies the release workflow and the two
# quickstart scripts into a scratch tree, asserts that the check passes them as they are, then
# breaks each rule in a fresh copy and asserts that the check fails, saying which: a job that
# does not run its script, or runs it only in a comment; a step that runs it gated by `if:` or
# allowed to fail by `continue-on-error:`, or a job that is; the archive attached before it is
# run; the twin's steps or commands not the script's; and either unpack step that no longer
# checks the archive's SHA-256, or checks it after unpacking.
set -uo pipefail
here=$(cd "$(dirname "$0")" && pwd)
repo=$(cd "$here/.." && pwd)
check="$here/check-wai-release-quickstart.py"
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
wf="$tmp/.github/workflows/wai-release.yml"
sh="$tmp/wai/quickstart/test/clean-machine.sh"
ps="$tmp/wai/quickstart/test/clean-machine.ps1"
mk() {
rm -rf "$tmp/.github" "$tmp/wai"
mkdir -p "$tmp/.github/workflows" "$tmp/wai/quickstart/test"
cp "$repo/.github/workflows/wai-release.yml" "$wf"
cp "$repo/wai/quickstart/test/clean-machine.sh" "$sh"
cp "$repo/wai/quickstart/test/clean-machine.ps1" "$ps"
}
fails=0
expect() { # expect <pass|fail> <what> [text the output must hold]
local want=$1 what=$2 needle=${3:-} out rc
out=$(python3 -I "$check" --root "$tmp" 2>&1); rc=$?
if [ "$want" = pass ] && [ $rc -ne 0 ]; then echo "FAIL $what: the check failed"; printf '%s\n' "$out" | sed 's/^/ /'; fails=$((fails+1)); return; fi
if [ "$want" = fail ] && [ $rc -eq 0 ]; then echo "FAIL $what: the check passed"; printf '%s\n' "$out" | sed 's/^/ /'; fails=$((fails+1)); return; fi
if [ -n "$needle" ] && ! grep -qF -- "$needle" <<<"$out"; then echo "FAIL $what: the output does not say '$needle'"; printf '%s\n' "$out" | sed 's/^/ /'; fails=$((fails+1)); return; fi
echo "ok $what"
}
# sub <file> <from> <to> [n]: replace the n-th (default the only) occurrence, which must exist.
sub() {
python3 - "$@" <<'P' || { echo "FAIL the fixture edit did not apply: $2"; fails=$((fails+1)); }
import sys
p, a, b = sys.argv[1:4]
n = int(sys.argv[4]) if len(sys.argv) > 4 else 0
s = open(p, encoding="utf-8").read()
if n == 0 and s.count(a) != 1 or s.count(a) < max(n, 1):
sys.exit(1)
i = -1
for _ in range(max(n, 1)):
i = s.index(a, i + 1)
open(p, "w", encoding="utf-8").write(s[:i] + b + s[i + len(a):])
P
}
mk; expect pass "the release and the two scripts as they are" "run the quickstart from the archive before attaching it"
# The jobs: each runs its script, in a step's command, ungated, before it attaches.
mk; sub "$wf" ' run: bash wai/quickstart/test/clean-machine.sh "$RUNNER_TEMP/dist/wai-${{ needs.version.outputs.version }}-${{ matrix.triple }}.tar.gz"' ' run: echo the quickstart is not run'
expect fail "the linux job runs no quickstart" "the linux job runs clean-machine.sh in no step's command"
mk; sub "$wf" ' bash wai/quickstart/test/clean-machine.sh "$RUNNER_TEMP/dist/wai-$v-${{ matrix.triple }}.tar.gz" $flag' ' # bash wai/quickstart/test/clean-machine.sh "$RUNNER_TEMP/dist/wai-$v-${{ matrix.triple }}.tar.gz" $flag'
expect fail "the macos job names the script in a comment alone" "the macos job runs clean-machine.sh in no step's command"
mk; sub "$wf" ' pwsh -NoProfile -File wai\quickstart\test\clean-machine.ps1' ' pwsh -NoProfile -File wai\quickstart\test\clean-machine-other.ps1'
expect fail "the windows job runs another script" "the windows job runs clean-machine.ps1 in no step's command"
mk; sub "$wf" ' - name: The quickstart from the archive, as a new user runs it
shell: bash
env:' ' - name: The quickstart from the archive, as a new user runs it
continue-on-error: true
shell: bash
env:'
expect fail "the macos step may fail without failing the job" "has \`continue-on-error:\`: it can be skipped or fail"
mk; sub "$wf" ' - name: The quickstart from the archive, as a new user runs it
shell: pwsh' ' - name: The quickstart from the archive, as a new user runs it
if: false
shell: pwsh'
expect fail "the windows step is skipped" "which runs clean-machine.ps1, has \`if:\`"
mk; sub "$wf" ' name: Linux · ${{ matrix.triple }}
' ' name: Linux · ${{ matrix.triple }}
continue-on-error: true
'
expect fail "the linux job may fail without failing the release" "the linux job has \`continue-on-error:\`"
mk; sub "$wf" ' - name: The quickstart from the archive, as a new user runs it
shell: bash
run: bash' ' - name: Attach first
shell: bash
run: bash wai/quickstart/publish.sh "$GITHUB_REF_NAME" "wai" "$RUNNER_TEMP"/dist/*
- name: The quickstart from the archive, as a new user runs it
shell: bash
run: bash'
expect fail "the linux job attaches the archive before running it" "comes after 'Attach first', which attaches the archive"
mk; sub "$wf" ' - name: Attach to the release
shell: pwsh' ' - name: Upload it
shell: pwsh
run: Invoke-RestMethod -Method Post -Uri "$env:FORGE_API/repos/x/releases/1/assets?name=a.zip"
- name: Attach to the release
shell: pwsh'
expect pass "an upload after the quickstart is in order"
mk; python3 - "$wf" <<'P'
import sys
p = sys.argv[1]
s = open(p, encoding="utf-8").read()
q = s.index(" - name: The quickstart from the archive, as a new user runs it\n shell: pwsh")
a = s.index(" - name: Attach to the release\n shell: pwsh")
e = s.index("\n formulae:", a)
open(p, "w", encoding="utf-8").write(s[:q] + s[a:e] + s[q:a] + s[e:])
P
expect fail "the windows job attaches its archive before it runs it" "which runs clean-machine.ps1, comes after 'Attach to the release'"
# The twin keeps the script's steps and commands.
mk; sub "$ps" "Step 'create' { wai create my-room --template video-room }" "Step 'create' { wai create my-room }"
expect fail "the twin runs another command" "the twin runs clean-machine.sh's"
mk; sub "$ps" "if (-not \$Failed) { Step 'verify' { wai verify wai-hello } }" ""
expect fail "the twin drops a step" "the twin runs clean-machine.sh's"
mk; sub "$sh" '[ -z "$failed" ] && step verify wai verify wai-hello' '[ -z "$failed" ] && step check wai verify wai-hello'
expect fail "the script's steps change" "its steps are"
# Each unpack step checks the archive's SHA-256, before it unpacks.
mk; sub "$sh" ' [ "$got" = "$want" ] || { echo "SHA-256 $got, not $want"; return 1; }' ' [ "$got" = "$want" ] || echo "SHA-256 $got, not $want"'
expect fail "the script's unpack step does not fail on a SHA-256 that differs" "clean-machine.sh: the unpack step does not fail when"
mk; sub "$sh" 'want=$(cut -d'"' '"' -f1 "$ARCHIVE.sha256")' 'want=$got'
expect fail "the script's unpack step does not read the SHA-256 beside the archive" 'clean-machine.sh: the unpack step does not read the SHA-256 beside the archive ("$ARCHIVE.sha256")'
mk; sub "$ps" ' if ($got -ne $want) { throw "SHA-256 $got, not $want" }' ' if ($got -ne $want) { Write-Output "SHA-256 $got, not $want" }'
expect fail "the twin's unpack step does not fail on a SHA-256 that differs" "clean-machine.ps1: the unpack step does not fail when"
mk; python3 - "$ps" <<'P'
import sys
p = sys.argv[1]
s = open(p, encoding="utf-8").read()
c = ' if ($got -ne $want) { throw "SHA-256 $got, not $want" }\n'
s = s.replace(c, "")
s = s.replace(" tar -xzf $Archive -C $Work\n }\n", " tar -xzf $Archive -C $Work\n }\n" + c, 1)
open(p, "w", encoding="utf-8").write(s)
P
expect fail "the twin checks the SHA-256 after unpacking" "clean-machine.ps1: the unpack step unpacks the archive before it checks its SHA-256"
if [ "$fails" -ne 0 ]; then
echo "check-wai-release-quickstart-selftest: $fails case(s) failed"
exit 1
fi
echo "check-wai-release-quickstart-selftest: every case held"

263
scripts/check-wai-release-quickstart.py Executable file
View file

@ -0,0 +1,263 @@
#!/usr/bin/env python3
"""check-wai-release-quickstart.py: the release runs the quickstart from each archive before it
attaches it, and the Windows twin keeps the script's steps.
wai-release.yml builds the quickstart archive on each platform and attaches it to the release.
Before it is attached, each archive runs the quickstart as a new user runs it, from the archive
alone: wai/quickstart/test/clean-machine.sh on macOS and Linux, its PowerShell twin
clean-machine.ps1 on Windows. Each step of the twin is a step of the script, with the same
`wai` command. This holds the three files to that:
- each of the jobs macos, linux and windows runs its script in a step's command (a comment or
a step's name that names it is not a run of it); that step has no `if:` and no
`continue-on-error:`, and neither has its job, so it can neither be skipped nor fail without
failing the job; and it comes before the job's first step that attaches anything to the
release (one that runs wai/quickstart/publish.sh or uploads to a release's assets);
- clean-machine.sh's steps are unpack, contents, hello, verify, create, dev-check and two-tabs
(the browser's), and clean-machine.ps1's are the same steps in the same order, the browser's
aside, each running the same `wai` command (the command alone, up to the first `;`, `}`,
`'`, `|`, `>` or `&&` after it);
- the unpack step of each checks the archive against the SHA-256 beside it ("$ARCHIVE.sha256",
"$Archive.sha256") and fails when they differ, before it unpacks anything.
check-wai-release-quickstart.py [--root DIR]
Exit 0 when all of it holds, 1 with one line a finding otherwise. Standard library only. The
selftest (check-wai-release-quickstart-selftest.sh) breaks each rule in a copy and requires a
finding.
"""
import os
import re
import sys
WORKFLOW = ".github/workflows/wai-release.yml"
SH = "wai/quickstart/test/clean-machine.sh"
PS = "wai/quickstart/test/clean-machine.ps1"
# The job, and the script it runs on its archive.
JOBS = [("macos", "clean-machine.sh"), ("linux", "clean-machine.sh"), ("windows", "clean-machine.ps1")]
SH_STEPS = ["unpack", "contents", "hello", "verify", "create", "dev-check", "two-tabs"]
BROWSER_STEP = "two-tabs"
def indent(line):
return len(line) - len(line.lstrip(" "))
def is_comment(line):
return line.strip().startswith("#")
def jobs(text):
"""{job name: its lines}, the jobs under the top-level `jobs:` key."""
out = {}
lines = text.splitlines()
try:
start = next(i for i, l in enumerate(lines) if l.rstrip() == "jobs:")
except StopIteration:
return out
name = None
for l in lines[start + 1:]:
if l.strip() and not is_comment(l) and indent(l) == 0:
break
m = re.match(r"^ ([A-Za-z0-9_-]+):\s*$", l)
if m:
name = m.group(1)
out[name] = []
elif name is not None:
out[name].append(l)
return out
def job_keys(lines):
"""The job's own keys (indented four spaces), comments aside."""
return [re.match(r"^\s*([A-Za-z0-9_-]+):", l).group(1) for l in lines if indent(l) == 4 and not is_comment(l) and re.match(r"^\s*([A-Za-z0-9_-]+):", l)]
def steps(lines):
"""The job's steps, in order: each {"keys": [its keys], "name": …, "run": [its command's
lines, comments aside]}. A step is a list item of `steps:`; its keys are the item's first
key and those at the indent of the item's content."""
out = []
in_steps = False
item_indent = None
cur = None
run_indent = None
for l in lines:
if not l.strip():
continue
if indent(l) == 4 and not is_comment(l):
in_steps = l.strip() == "steps:"
cur = None
continue
if not in_steps:
continue
s = l.lstrip(" ")
if s.startswith("- ") and not is_comment(l) and (item_indent is None or indent(l) == item_indent):
item_indent = indent(l)
cur = {"keys": [], "name": "", "run": []}
out.append(cur)
run_indent = None
l = " " * (item_indent + 2) + s[2:]
if cur is None:
continue
key_indent = item_indent + 2
if run_indent is not None:
if indent(l) > key_indent:
if not is_comment(l):
cur["run"].append(l.strip())
continue
run_indent = None
if is_comment(l) or indent(l) != key_indent:
continue
m = re.match(r"^\s*([A-Za-z0-9_-]+):\s*(.*)$", l)
if not m:
continue
key, value = m.group(1), m.group(2).strip()
cur["keys"].append(key)
if key == "name":
cur["name"] = value.strip("'\"")
elif key == "run":
if value in ("|", ">", "|-", ">-", "|+", ">+"):
run_indent = key_indent
elif value and not value.startswith("#"):
cur["run"].append(value)
return out
def runs_script(step, script):
"""Whether the step's command runs `script` (wai/quickstart/test/<script>, either slash)."""
want = (f"wai/quickstart/test/{script}", f"wai\\quickstart\\test\\{script}")
return any(w in l.split(" #")[0] for l in step["run"] for w in want)
def attaches(step):
"""Whether the step attaches something to the release."""
return any("wai/quickstart/publish.sh" in l or re.search(r"/assets\b", l) for l in step["run"])
def check_workflow(text, findings):
js = jobs(text)
for job, script in JOBS:
if job not in js:
findings.append(f"{WORKFLOW}: no job {job}, which runs {script} on the archive it attaches")
continue
where = f"{WORKFLOW}: the {job} job"
for k in ("if", "continue-on-error"):
if k in job_keys(js[job]):
findings.append(f"{where} has `{k}:`, so the quickstart from its archive can be skipped or fail without failing it")
st = steps(js[job])
running = [i for i, s in enumerate(st) if runs_script(s, script)]
if not running:
findings.append(f"{where} runs {script} in no step's command: the archive it attaches is not run as a new user runs it")
continue
attach = next((i for i, s in enumerate(st) if attaches(s)), None)
if attach is None:
findings.append(f"{where} has no step that attaches its archive to the release (wai/quickstart/publish.sh, or an upload to the release's assets)")
for i in running:
s = st[i]
what = f"{where}'s step {s['name']!r}, which runs {script},"
for k in ("if", "continue-on-error"):
if k in s["keys"]:
findings.append(f"{what} has `{k}:`: it can be skipped or fail without failing the job")
if attach is not None and i > attach:
findings.append(f"{what} comes after {st[attach]['name']!r}, which attaches the archive: the archive is attached before it is run")
def wai_command(rest):
"""The `wai` command in a step's text, alone: from `wai ` up to the first `;`, `}`, `'`,
`|`, `>` or `&&` after it (what the step does with its output is not the command); None
when the step runs no `wai` command."""
i = rest.find("wai ")
return re.split(r"[;}'|>]|&&", rest[i:])[0].strip() if i >= 0 else None
def sh_steps(text):
"""clean-machine.sh's steps: (name, the `wai` command it runs, or None)."""
out = []
for l in text.splitlines():
t = l.strip()
m = re.match(r"^(?:\[[^]]*\]\s*&&\s*)*step ([A-Za-z0-9-]+)(?: (.*))?$", t)
if not m:
continue
out.append((m.group(1), wai_command(m.group(2) or "")))
return out
def ps_steps(text):
"""clean-machine.ps1's steps: (name, the `wai` command its block runs, or None)."""
out = []
for l in text.splitlines():
m = re.search(r"\bStep '([A-Za-z0-9-]+)' \{(.*)$", l.strip())
if not m:
continue
out.append((m.group(1), wai_command(m.group(2))))
return out
def block(text, start_re):
"""The lines of the block that starts on the line matching `start_re` and ends at the next
line that is `}` alone at the start of a line."""
lines = text.splitlines()
for i, l in enumerate(lines):
if re.search(start_re, l):
for j in range(i + 1, len(lines)):
if lines[j].rstrip() == "}":
return lines[i + 1:j]
return lines[i + 1:]
return None
def check_unpack(name, body, sums, compare, unpack, findings):
"""The unpack step's body reads the SHA-256 from `sums`, fails when it differs (`compare`),
and does both before it unpacks (`unpack`)."""
if body is None:
findings.append(f"{name}: no unpack step")
return
code = [l.strip() for l in body if not l.strip().startswith("#")]
reads = next((i for i, l in enumerate(code) if sums in l), None)
fails = next((i for i, l in enumerate(code) if re.search(compare, l)), None)
opens = next((i for i, l in enumerate(code) if re.search(unpack, l)), None)
if reads is None:
findings.append(f"{name}: the unpack step does not read the SHA-256 beside the archive ({sums})")
if fails is None:
findings.append(f"{name}: the unpack step does not fail when the archive's SHA-256 is not the one beside it")
if opens is None:
findings.append(f"{name}: the unpack step unpacks nothing")
elif any(x is not None and x > opens for x in (reads, fails)):
findings.append(f"{name}: the unpack step unpacks the archive before it checks its SHA-256")
def check_scripts(sh, ps, findings):
s, p = sh_steps(sh), ps_steps(ps)
names = [n for n, _ in s]
if names != SH_STEPS:
findings.append(f"{SH}: its steps are {names}, not {SH_STEPS}")
want = [x for x in s if x[0] != BROWSER_STEP]
if p != want:
findings.append(f"{PS}: its steps are {p}; the twin runs clean-machine.sh's, the browser's aside: {want}")
check_unpack(SH, block(sh, r"^unpack\(\)\s*\{"), '"$ARCHIVE.sha256"', r'\[ "\$got" = "\$want" \] \|\| \{.*\breturn 1', r"\btar -x", findings)
check_unpack(PS, block(ps, r"^Step 'unpack' \{"), '"$Archive.sha256"', r"if \(\$got -ne \$want\) \{\s*throw\b", r"\bExpand-Archive\b|\btar -x", findings)
def main(argv):
root = "."
if argv[:1] == ["--root"] and len(argv) == 2:
root = argv[1]
elif argv:
print("usage: check-wai-release-quickstart.py [--root DIR]", file=sys.stderr)
return 2
read = lambda p: open(os.path.join(root, p), encoding="utf-8").read()
findings = []
check_workflow(read(WORKFLOW), findings)
check_scripts(read(SH), read(PS), findings)
for f in findings:
print(f"FAIL {f}")
if not findings:
print(f"ok {WORKFLOW}: the macos, linux and windows jobs run the quickstart from the archive before attaching it")
print(f"ok {PS} keeps {SH}'s steps, and both check the archive's SHA-256 before unpacking it")
print(f"check-wai-release-quickstart: {len(findings)} finding(s)")
return 1 if findings else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))

15
scripts/wai-conformance-lists-selftest.sh
View file

@ -206,6 +206,21 @@ expect 1 "a step file with no body" "no body" -- check
mk; printf '# corpus: x\n# order: 1\n# bin: wai_world_vectors\n' > "$T/.github/wai-conformance/corpora/x.txt"
expect 1 "a corpus file with no directory" "no \`dir:\` line" -- check
# The independent verifiers: one `independent:` line an entry, run by the union step alone.
mk
expect 0 "a declaration, entries and no body, is no step" "a declaration; its verifiers run in the union step" -- run --only loc
expect 0 "the entries print in order, each with its command line" "loc live python3 -I live-conformance/loc_verify.py live-conformance/loc" -- independent
mk; ed "$V/loc.sh" 's/^# independent: loc live /# independent: loc nowhere /'
expect 1 "an entry of no area" "is none of trust, media, delivery, live" -- check
mk; ed "$V/loc.sh" 's/^# independent: loc live python3 /# independent: loc live /'
expect 1 "an entry with no command line" "before \`python3\`" -- check
mk; ed "$V/loc.sh" 's/^# independent: loc live /# independent: e2ee live /'
expect 1 "an entry's name another file gives" "is also" -- check
mk; printf 'python3 -I live-conformance/loc_verify.py live-conformance/loc\n' >> "$V/loc.sh"
expect 1 "a body that runs an entry's command line itself" "itself; the union step runs it" -- check
mk; perl -ni -e 'print unless /^# independent: /' "$V/loc.sh"
expect 1 "a declaration whose entries are taken away" "no body" -- check
# The commands refuse what they cannot answer, rather than answering with nothing.
mk
expect 1 "a tag no step takes, asked for" "a tag is one of" -- tests int-kernel

148
scripts/wai-conformance-lists.py
View file

@ -38,6 +38,13 @@ A verifier file's header (the leading `# key: value` lines; the first other line
/pq, beside every `wai-bins/union/NAME` and `wai-bins/pq/NAME` the body names
paths globs whose change runs the workflow (each under one of its `paths:` globs)
working-directory where the body runs, from the repository root (default wai)
independent one line an entry: a corpus's independent verifier, which the job runs with
every other in one step, `wai conform --independent`, and never in a body:
`<name> <area> [needs=<packages>] [verdicts=<file>] [dir=<dir>] python3 [-I]
<script> <args>`, the command line run in wai/ as the corpus's README gives
it. `wai` builds its table of independent verifiers from these lines
(wai-rs/build.rs). A file whose header holds entries and that has no body is a
declaration: its verifiers run in the union alone, and it is no step of its own.
The tests the job builds first are the `--test NAME` of the body's lines that build with
"$WAI_FEATURES", and with "$WAI_RELEASE_FEATURES" for the optimised build.
@ -47,7 +54,8 @@ Commands (the workflow runs the first group):
env WAI_FEATURES=..., WAI_PQ_FEATURES=..., WAI_RELEASE_FEATURES=...
prebuild the shell script that builds the binaries and tests first
run [--only SLUG]... every verifier step in order, each in a log group; the first that
fails stops the run with its exit status
fails stops the run with its exit status (a declaration has no step)
independent the independent verifiers' entries, in order: `name area command` a line
alone the feature lists features-alone checks, one a line
paths the paths the verifier files bring to the gate, one a line
tests TAG... `--test NAME` for every test file that carries one of the tags
@ -103,7 +111,7 @@ TAGS = ("int-kernels", "int-kernels-release", "int-kernels-amd")
VERIFIER_KEYS = {
"step": "one", "order": "one", "features": "many", "alone": "many", "build": "one",
"release-features": "many", "bins": "many", "pq-bins": "many", "paths": "many",
"working-directory": "one",
"working-directory": "one", "independent": "lines",
}
CORPUS_KEYS = {"corpus": "one", "order": "one", "bin": "one", "dir": "one"}
HEADER = re.compile(r"^# ([a-z][a-z-]*): ?(.*)$")
@ -155,6 +163,8 @@ def read_header(path, keys, label):
if k in head:
errs.append(f"{label}: line {i + 1}: `{k}` is given twice")
head[k] = v
elif keys[k] == "lines":
head.setdefault(k, []).append(v)
else:
head.setdefault(k, []).extend(v.split())
i += 1
@ -164,6 +174,77 @@ def read_header(path, keys, label):
return head, body, errs
AREAS = ("trust", "media", "delivery", "live")
ENTRY_NAME = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*$")
def independent_entry(line):
"""(entry, None) of an `independent:` line, or (None, why it is not one). The same reading
as wai-rs/src/bin/wai_cli/dx/independent_table.rs, which build.rs runs."""
w = line.split()
if len(w) < 4:
return None, "an entry is `<name> <area> [key=value]... python3 [-I] <script> <args>`"
name, area, rest = w[0], w[1], w[2:]
if not ENTRY_NAME.match(name) or name in AREAS:
return None, f"the name `{name}` is lowercase words joined by hyphens, and no area's"
if area not in AREAS:
return None, f"the area `{area}` is none of {', '.join(AREAS)}"
opts = {}
while rest and rest[0] != "python3":
k, eq, v = rest[0].partition("=")
if not eq or k not in ("needs", "verdicts", "dir") or not v or k in opts:
return None, f"`{rest[0]}`: an option is needs=, verdicts= or dir=, each once, before `python3`"
opts[k] = v
rest = rest[1:]
if not rest:
return None, "no command line: it starts with `python3`"
rest = rest[1:]
isolated = bool(rest) and rest[0] == "-I"
if isolated:
rest = rest[1:]
if not rest:
return None, "the command line names no script"
return {"name": name, "area": area, "needs": opts.get("needs", "").split(",") if "needs" in opts else [],
"verdicts": opts.get("verdicts"), "dir": opts.get("dir"), "isolated": isolated,
"script": rest[0], "args": rest[1:],
"command": " ".join(["python3"] + (["-I"] if isolated else []) + rest)}, None
def entry_dir(e, wai):
"""The corpus directory of an entry: its `dir=`, or the first argument that names a
directory under wai/, or the directory of the first that names a file."""
if e["dir"]:
return e["dir"]
for a in e["args"]:
if a.startswith("-"):
continue
p = os.path.join(wai, a)
if os.path.isdir(p):
return a
if os.path.isfile(p):
return posix_dirname(a)
return None
def posix_dirname(p):
return p.rsplit("/", 1)[0] if "/" in p else ""
def runs_entry(line, e):
"""Whether a body's line runs an entry's command line itself (from wai/ or the root)."""
w = line.strip().split()
if not w or not w[0].startswith("python"):
return False
w = w[1:]
if w and w[0] == "-I":
w = w[1:]
norm = lambda x: x[4:] if x.startswith("wai/") else x
if not w or norm(w[0]) != e["script"]:
return False
args = [norm(x) for x in w[1:]]
return args[:len(e["args"])] == e["args"] and (len(args) == len(e["args"]) or args[len(e["args"])] in (">", "|", ">>", "2>&1", "&&", ";"))
def joined(body):
"""The body's lines with `\\` continuations joined."""
return re.sub(r"\\\n\s*", " ", body).split("\n")
@ -196,7 +277,14 @@ def verifiers(root):
alone = []
elif alone is not None and "-" in alone:
errs.append(f"{rel}: alone `-` stands alone")
if not body.strip():
entries = []
for line in h.get("independent", []):
e, why = independent_entry(line)
if why:
errs.append(f"{rel}: independent `{line}`: {why}")
else:
entries.append(e)
if not body.strip() and not entries:
errs.append(f"{rel}: no body: the step would run nothing")
mods.append({
"slug": f[:-3], "file": rel, "step": h.get("step", ""), "order": order,
@ -204,6 +292,7 @@ def verifiers(root):
"release": h.get("release-features", []), "bins": h.get("bins", []),
"pq_bins": h.get("pq-bins", []), "paths": h.get("paths", []),
"wd": h.get("working-directory", "wai"), "body": body,
"independent": entries, "declaration": not body.strip(),
})
mods.sort(key=lambda m: (m["order"], m["slug"]))
seen = {}
@ -391,7 +480,9 @@ def from_tree(root):
"union_bins": d["union_bins"], "pq_bins": d["pq_bins"],
"union_tests": d["union_tests"], "release_tests": d["release_tests"],
"alone": d["alone"],
"steps": [{"step": m["step"], "wd": m["wd"], "body": m["body"]} for m in mods],
"steps": [{"step": m["step"], "wd": m["wd"], "body": m["body"]} for m in mods if not m["declaration"]],
"declarations": [m["step"] for m in mods if m["declaration"]],
"independent": [f"{e['name']} {e['area']} {e['command']}" for m in mods for e in m["independent"]],
"int_kernels_debug_features": tagged_features(root, ["int-kernels"], x.get("test-tagged int-kernels", [])),
"int_kernels_debug_tests": dbg,
"int_kernels_release_features": tagged_features(root, ["int-kernels", "int-kernels-release"], x.get("test-tagged int-kernels int-kernels-release", [])),
@ -789,6 +880,12 @@ def dry_run(L, C):
out.append(f"{k}: {' '.join(norm_list(k, L[k]))}")
out.append("corpora (in order):")
out += [f" {c}" for c in L["corpora"]]
if L.get("independent"):
out.append("independent verifiers, run by one step as `wai conform --independent` (in order):")
out += [f" {x}" for x in L["independent"]]
if L.get("declarations"):
out.append("declarations (step files whose verifiers run in the union alone):")
out += [f" {x}" for x in L["declarations"]]
out.append("independent-verifiers steps (in order):")
for s in L["steps"]:
out.append(f" - step: {s['step']} (working-directory: {s['wd']})")
@ -844,6 +941,30 @@ def check(root):
errs.append(f"{m['file']}: path `{p}` is under none of {WORKFLOW}'s paths, so the forge would never start the workflow for it")
if full and not os.path.isdir(os.path.join(root, m["wd"])):
errs.append(f"{m['file']}: working-directory `{m['wd']}` does not exist")
seen = {}
every = [(m, e) for m in mods for e in m["independent"]]
for m, e in every:
if e["name"] in seen:
errs.append(f"{m['file']}: independent `{e['name']}` is also {seen[e['name']]}'s")
seen.setdefault(e["name"], m["file"])
if full:
wai = os.path.join(root, "wai")
if not os.path.isfile(os.path.join(wai, e["script"])):
errs.append(f"{m['file']}: independent `{e['name']}`: wai/{e['script']} is not there")
for a in e["args"]:
if "/" in a and not a.startswith("-") and not os.path.exists(os.path.join(wai, a)):
errs.append(f"{m['file']}: independent `{e['name']}`: wai/{a} is not there")
d = entry_dir(e, wai)
if not d or not os.path.isdir(os.path.join(wai, d)):
errs.append(f"{m['file']}: independent `{e['name']}`: no corpus directory (give dir=)")
if e["verdicts"] and not os.path.isfile(os.path.join(wai, e["verdicts"])):
errs.append(f"{m['file']}: independent `{e['name']}`: verdicts file wai/{e['verdicts']} is not there")
for m in mods:
for line in joined(m["body"]):
for o, e in every:
if runs_entry(line, e):
errs.append(f"{m['file']}: runs the independent verifier `{e['name']}` ({o['file']}) itself; "
f"the union step runs it, `wai conform --independent`")
for c in cs:
if c["bin"] not in bins:
errs.append(f"{c['file']}: binary `{c['bin']}` is not a [[bin]] of {CRATE}/Cargo.toml")
@ -881,8 +1002,9 @@ def selftest(root, exact):
else:
print(f"ok {k}: the hand list's {len(want)}{' and ' + str(len(more)) + ' more' if more else ''}")
names = [s["step"] for s in L["steps"]]
declared = L.get("declarations", [])
for s in base["steps"]:
if s["step"] not in names:
if s["step"] not in names and s["step"] not in declared:
print(f"FAIL step `{s['step']}` no longer runs")
bad += 1
if exact:
@ -905,7 +1027,9 @@ def selftest(root, exact):
if [c for c in L["corpora"] if c in base["corpora"]] != [c for c in base["corpora"] if c in L["corpora"]]:
print("FAIL the corpora both lists hold run in another order than the hand list's")
bad += 1
print(f"ok {len(base['steps'])} steps of the hand list run" if not bad else f"wai-conformance-lists selftest: {bad} failure(s)")
through = sum(1 for s in base["steps"] if s["step"] in declared)
print(f"ok {len(base['steps'])} steps of the hand list run" + (f" ({through} of them as declarations whose verifiers the union runs)" if through else "")
if not bad else f"wai-conformance-lists selftest: {bad} failure(s)")
return 1 if bad else 0
@ -929,7 +1053,7 @@ def main(argv):
return 2
cmd, rest = args[0], args[1:]
try:
if cmd in ("env", "prebuild", "run", "alone", "paths"):
if cmd in ("env", "prebuild", "run", "alone", "paths", "independent"):
mods, errs = verifiers(root)
if errs:
raise Bad("\n".join(errs))
@ -943,6 +1067,10 @@ def main(argv):
print("\n".join(d["alone"]))
elif cmd == "paths":
print("\n".join(d["paths"]))
elif cmd == "independent":
for m in mods:
for e in m["independent"]:
print(f"{e['name']} {e['area']} {e['command']}")
else:
only = [rest[i + 1] for i, a in enumerate(rest) if a == "--only"]
env = dict(os.environ)
@ -952,6 +1080,10 @@ def main(argv):
todo = [m for m in mods if not only or m["slug"] in only]
if only and len(todo) != len(set(only)):
raise Bad(f"--only names a step no file holds: {' '.join(sorted(set(only) - {m['slug'] for m in mods}))}")
for m in [m for m in todo if m["declaration"]]:
print(f"{m['step']} ({m['file']}): a declaration; its verifiers run in the union step, "
f"`wai conform --independent`: {', '.join(e['name'] for e in m['independent'])}", flush=True)
todo = [m for m in todo if not m["declaration"]]
for m in todo:
print(f"::group::{m['step']} ({m['file']})", flush=True)
rc = subprocess.call(["bash", "--noprofile", "--norc", "-eo", "pipefail", os.path.join(root, m["file"])],
@ -960,7 +1092,7 @@ def main(argv):
if rc:
print(f"::error::{m['step']} failed with exit status {rc} ({m['file']})", flush=True)
return rc
print(f"every step ran and passed: {len(todo)} of {len(mods)}")
print(f"every step ran and passed: {len(todo)} of {sum(1 for m in mods if not m['declaration'])}")
elif cmd in ("tests", "features"):
print(script_value(root, [cmd] + rest))
elif cmd == "test-tagged":

10
wai/CONTRIBUTING.md
View file

@ -116,9 +116,13 @@ inserts beside the entry it builds on:
workflow's gate runs on (the trigger takes all of `extensions/`; a path under none of the
trigger's paths still needs a line there, which `check` names); `# bins:` and
`# release-features:` cover the rest, and every `--test` it runs with `"$WAI_FEATURES"`
joins the job's first build. `scripts/wai-conformance-lists.py`
states every key; `python3 scripts/wai-conformance-lists.py dry-run` prints every list
and step the workflow runs, and `run --only <name>` runs one step.
joins the job's first build. A corpus's independent verifier is an `# independent:` line
of its step's file (its name, its area, and the command line its README gives), never a
line of the body: one step runs every one of them as `wai conform --independent`, whose
table `wai-rs/build.rs` reads from those lines. A step that has only verifiers to run is
a file with that header and no body. `scripts/wai-conformance-lists.py` states every key;
`python3 scripts/wai-conformance-lists.py dry-run` prints every list and step the
workflow runs, and `run --only <name>` runs one step.
- **A cargo-class corpus** is a new file of `.github/wai-conformance/corpora/`, naming
its tool and directory. **A test of the fast integer kernels** carries the first line
`// wai-conformance: int-kernels` (with `int-kernels-amd` when the AVX-512 host runs it

12
wai/README.md
View file

@ -75,6 +75,7 @@ wai hello # name, dispatch, rebuild and receip
wai create my-room --template video-room # a project from one of the examples (examples/)
cd my-room && wai dev # the relay, the room service and your page, on 127.0.0.1:8080
wai verify ../wai-hello # the receipts hello wrote, checked offline
wai doctor # which wai runs when you type it, and what wai dev needs
```
[`examples/`](examples/README.md) holds each template, with a test that runs it.
@ -103,14 +104,19 @@ to WAI's corpora:
wai breaking --against origin/main # what this tree breaks: wire, record and API (CI runs it on every pull request)
wai conform verifier --suite claim --impl "my-verifier verdicts {corpus}" # a corpus run against your implementation
wai verify claim.wai-conform.json # the signed result, checked
wai conform --independent --out records --format junit # every corpus's independent verifier, a record each
```
A break passes only when the changelog names it: an entry this change adds, as a file of
[`changes/`](changes/) or to [`CHANGELOG.md`](CHANGELOG.md), that says **Breaking** and gives the
finding's id in backticks. `wai conform --list` names every kind
(verifier, decoder, packager, origin, sink) and suite, with its adapters: a command line, or an
`http://` URL. `wai verify --conformant` admits a result only for a run that passed, against the
corpus of the checkout it runs in.
`http://` URL; and every corpus's independent verifier, its second implementation, with the
command line it runs. `--independent` beside a kind runs the corpus's independent verifier on
the same cases, and the run passes only when both conform and agree on every case. `--format
junit` or `--format html` prints a report made from the signed records. `wai verify
--conformant` admits a result only for a run that passed, against the corpus of the checkout it
runs in.
With the `codecs` feature, `wai-rs/build.rs` finds libFLAC through pkg-config
and, on macOS without pkg-config, in the Homebrew "flac" formula prefix
@ -135,7 +141,7 @@ operation of the toolchain, named by it with `_` for the dot, and the tools come
| Toolset | Tools |
|---|---|
| `core` (the default) | `envelope_inspect`, `media_inspect`, `registry_list`, `registry_show`, `receipt_verify`, `envelope_wrap`, `conformance_list`, `conformance_run`, `docs_search` |
| `core` (the default) | `envelope_inspect`, `media_inspect`, `registry_list`, `registry_show`, `receipt_verify`, `envelope_wrap`, `conformance_list`, `conformance_run`, `toolchain_doctor`, `docs_search` |
| `dev` | `dev_start`, `dev_stop`, `room_grant`, `room_join`, `room_leave`, `session_show` |
| `app` (a build with the feature `app`) | `app_guide`, `app_check`, `app_stream`, `app_export`: Create's, below |

10
wai/catalogue/README.md
View file

@ -14,7 +14,7 @@ An operation is named `noun.verb`: lowercase ASCII, a singular noun, and a verb
rule N3's closed list (create, list, show, update, delete, check, verify, run, build, export,
import, start, stop, join, leave, grant, invite, revoke, rotate, tail, search, test, load, encode,
decode, inspect, wrap, unwrap, fetch, generate, promote, rollback, lock, upgrade, setup, guide,
stream, upscale, tonemap), never a near-synonym (`get`, `describe` and `info` are `show`; `keygen`, `init` and
stream, upscale, tonemap, doctor), never a near-synonym (`get`, `describe` and `info` are `show`; `keygen`, `init` and
`new` are `create`; `up` is `start`, as in `wai dev`). A command a person already types keeps its spelling as the operation's
command line: `wai hello` is `toolchain.test`, `wai stage refine` is `stage.encode`.
@ -23,9 +23,11 @@ The list as approved with WAI toolchain v1 ended at `stream`. Two verbs joined i
exact derivation, an output any verifier recomputes bit for bit from the input and the parameters
its record names (extensions/exact-upscale.md, extensions/exact-tonemap.md), and no approved verb
names that: `encode` and `decode` run a codec, and `generate` makes media from a description. A
verb joins the list the same way: in `VERBS` (wai-rs/src/catalogue.rs), this paragraph and the
list above, and the doors' check (`scripts/check-wai-ops-doors.py`), which a test of the catalogue
holds to `VERBS`.
third, `doctor`, joined with `toolchain.doctor` (`wai doctor`), which the specification's own
operation table names. A verb joins the list the same way: in `VERBS` (wai-rs/src/catalogue.rs),
this paragraph and the list above. The doors' check (`scripts/check-wai-ops-doors.py`) keeps no
list of its own: it reads `VERBS` and `NEAR_SYNONYMS` from that file, and a test of the
catalogue holds this paragraph's lists to them.
An entry (format `wai.ops/1`) carries:

106
wai/catalogue/ops.json
View file

@ -130,6 +130,57 @@
"sdk": "wai.toolchain.test"
}
},
{
"id": "toolchain.doctor",
"summary": "Which wai runs when you type wai, this install and what wai dev runs beside it, the loopback ports, browsers and the keychain, checked",
"cli": "wai doctor",
"aliases": [],
"usage": [
"wai doctor which `wai` runs when you type wai (first: another program of that name on PATH, such as the one",
" the crates.io package wai-cli installs), this install and what `wai dev` runs beside it, the loopback ports it",
" listens on, the browsers registered to open web pages, and where a credential is kept; exit 1 when a check fails"
],
"placement": "local",
"effect": "reads",
"dry_run": false,
"confirm": "never",
"route": null,
"doors": {
"cli": true,
"mcp": true,
"http": false
},
"toolset": "core",
"stability": "preview",
"input": {
"type": "object",
"properties": {},
"required": [],
"additionalProperties": false
},
"output": {
"type": "object",
"required": [
"wai",
"checks",
"failed",
"warnings"
],
"description": "Each check in order (path, install, ports, browsers, keychain) with its status (ok, warn or fail), what it found and what to do"
},
"stream": null,
"since": "0.3.0",
"deprecated": null,
"receipt": false,
"needs": null,
"names": {
"cli": "wai toolchain doctor",
"mcp": "toolchain_doctor",
"c": "wai_toolchain_doctor",
"rust": "wai::toolchain::doctor",
"sdk": "wai.toolchain.doctor"
}
},
{
"id": "project.create",
"summary": "A new project from one of the examples, written into a new or empty directory; --list names the templates",
@ -3457,13 +3508,18 @@
},
{
"id": "conformance.run",
"summary": "WAI's conformance corpus run against another implementation, through a command or an HTTP adapter, its result a signed record wai verify checks",
"summary": "WAI's conformance corpus run against another implementation, through a command or an HTTP adapter, or every corpus's independent verifier; each result a signed record wai verify checks",
"cli": "wai conform",
"aliases": [],
"usage": [
"wai conform <kind> --suite <suite> --impl <command or URL> [--key <file>] [--out <file>] [--root <wai-dir>]",
" [--timeout <seconds>] [--name <text>] [--version <text>] [--all] [--yes] `wai conformance list` names",
" the kinds and suites; a record that exists is replaced only with --yes (--confirm, deprecated, is the same)"
"wai conform <kind> --suite <suite> --impl <command or URL> [--independent] [--key <file>] [--out <file>]",
" [--root <wai-dir>] [--timeout <seconds>] [--name <text>] [--version <text>] [--python <program>] [--all]",
" [--format text|json|junit|html] [--yes] `wai conformance list` names the kinds and suites; --independent",
" runs the corpus's independent verifier beside the implementation, and both must conform",
"wai conform --independent [--suite <area|name>[,...]] [--out <dir>] [--key <file>] [--root <wai-dir>]",
" [--timeout <seconds>] [--python <program>] [--format text|json|junit|html] [--yes] every corpus's",
" independent verifier (or those named), a record each in <dir>; a record that exists is replaced only with",
" --yes (--confirm, deprecated, is the same)"
],
"placement": "local",
"effect": "writes-project",
@ -3482,29 +3538,36 @@
"properties": {
"kind": {
"type": "string",
"description": "The kind of implementation (wai conformance list): a decoder, a verifier, an origin, a packager, a sink.",
"description": "The kind of implementation (wai conformance list): a decoder, a verifier, an origin, a packager, a sink; none with --independent alone.",
"x-cli": {
"positional": true
}
},
"suite": {
"type": "string",
"description": "The suite (wai conformance list).",
"description": "The suite (wai conformance list); with --independent alone, areas and independent verifiers, comma-separated.",
"x-cli": {
"flag": "--suite"
}
},
"impl": {
"type": "string",
"description": "The implementation: a command with {case}/{in}/{out}/{corpus} placeholders, or a URL.",
"description": "The implementation: a command with {case}/{in}/{out}/{corpus} placeholders, or a URL; required with a kind.",
"x-cli": {
"flag": "--impl"
}
},
"independent": {
"type": "boolean",
"description": "Run the corpus's independent verifier beside the implementation, held to the same cases; without a kind, every independent verifier.",
"x-cli": {
"flag": "--independent"
}
},
"key": {
"type": "string",
"format": "path",
"description": "The seed the record is signed with.",
"description": "The seed the records are signed with.",
"x-cli": {
"flag": "--key"
}
@ -3512,7 +3575,7 @@
"out": {
"type": "string",
"format": "path",
"description": "The record (<suite>.wai-conform.json).",
"description": "The record (<suite>.wai-conform.json); with --independent alone, a directory for a record a corpus.",
"x-cli": {
"flag": "--out"
}
@ -3527,7 +3590,7 @@
},
"timeout": {
"type": "integer",
"description": "Seconds a case may take.",
"description": "Seconds a case may take (60), or an independent verifier its whole corpus (1800).",
"x-cli": {
"flag": "--timeout"
}
@ -3546,6 +3609,20 @@
"flag": "--version"
}
},
"python": {
"type": "string",
"description": "The Python the independent verifiers run with (python3).",
"x-cli": {
"flag": "--python"
}
},
"format": {
"type": "string",
"description": "What is printed: text, json (the envelope, as --json), junit (JUnit XML) or html, the last two from the records.",
"x-cli": {
"flag": "--format"
}
},
"all": {
"type": "boolean",
"description": "List every case, passed ones too.",
@ -3554,10 +3631,7 @@
}
}
},
"required": [
"kind",
"impl"
],
"required": [],
"additionalProperties": false
},
"output": {
@ -3588,11 +3662,11 @@
},
{
"id": "conformance.list",
"summary": "The kinds and suites wai conform runs, each suite's corpus and its cases",
"summary": "The kinds and suites wai conform runs, each suite's corpus and its cases, and every corpus's independent verifier",
"cli": "wai conformance list",
"aliases": [],
"usage": [
"wai conformance list [--root <wai-dir>] (also wai conform --list)"
"wai conformance list [--root <wai-dir>] the kinds and suites, and the independent verifiers (also wai conform --list)"
],
"placement": "local",
"effect": "reads",

21
wai/changes/2026-11-04-independent-verifiers-table-from-step-files.md Normal file
View file

@ -0,0 +1,21 @@
### The table of independent verifiers is read from the conformance workflow's step files
- **No list to append to.** `wai conform --independent`'s table is not kept by hand in `dx/independent.rs`:
`wai-rs/build.rs` reads it from the `# independent:` lines of the step files of `wai-conformance.yml`'s independent
verifiers (`.github/wai-conformance/verifiers/*.sh`), one line an entry, `<name> <area> [needs=<packages>]
[verdicts=<file>] [dir=<dir>] python3 [-I] <script> <args>`: the command line the corpus's README gives. A corpus's
verifier joins the union by a line in its own step's file, which no other change edits, and the entries run in the
files' order. A step file whose verifiers run in the union alone keeps its header and prose and has no body (a
declaration). `scripts/wai-conformance-lists.py` reads the same lines: `check` refuses a line it cannot read, a name
given twice, a script or corpus that is not there and a step that runs an entry itself, and `dry-run` and
`independent` print the entries.
- **A build without the step files has no table, and says so.** A `wai` built outside the repository, or from step
files it could not read, refuses `wai conform --independent` and `--independent` beside a kind with the reason
(`no-independent-table`), and `wai conform --list` says it has none, rather than running nothing and passing.
- The table's tests read the step files, and `wai-breaking.yml`, which runs them, runs on a change to one: the table
compiled in is their lines as they are now, and no step file's body runs an entry's command line. A verifier that a step's body runs itself is covered by that step, as the opus deep
redundancy and header probe steps run theirs over the inputs they fetch; a step whose verifier has no
`# independent:` line keeps running it in its body, so a change that adds one merges with this one in either order,
and the verifier joins the union by that line. The catalogue-prior record's verifier,
`wai_catalogue_prior_verify.py`, joins the union as the determinism-cost record's did, and the native sink suite names
its corpus's verifier for `--independent`.

13
wai/changes/2026-11-04-release-runs-quickstart-from-each-archive.md Normal file
View file

@ -0,0 +1,13 @@
### The release runs the quickstart from each archive as a new user runs it
- `wai-release.yml` runs `wai/quickstart/test/clean-machine.sh` on each macOS and Linux archive before attaching it: the
archive checked against its SHA-256 and unpacked on a fresh home directory and a PATH of the system's own directories,
then `wai hello`, `wai verify`, `wai create my-room --template video-room` and `wai dev --check` in the project; on the
Apple-silicon Mac, also the project's page joined in two browser pages, each decoding the other.
- `clean-machine.ps1` is its PowerShell twin for the Windows archive, step for step.
`scripts/check-wai-release-quickstart.py`, in the repository checks every change runs, holds each leg to running its
script before the archive is attached, in a step that is neither skipped (`if:`) nor allowed to fail
(`continue-on-error:`), the twin to the script's steps and commands, and both unpack steps to checking the archive's
SHA-256 before unpacking it. **The Windows leg has not run yet**: its runner is down, so no Windows archive has been
through it. Under PowerShell 7 on a macOS archive it has run without its check of what the archive holds and of
WAI Studio, which have not run yet.

35
wai/changes/2026-11-04-wai-conform-reports-ci-system-reads.md Normal file
View file

@ -0,0 +1,35 @@
### `wai conform`: reports a CI system reads, the independent verifiers beside an implementation, and the corpus union as one command
- **`--format text|json|junit|html`** (operation `conformance.run`). `junit` prints a JUnit XML report, a test suite a
signed record and a test case a case, a failure exactly where a case failed, with the record's corpus digest,
implementation and signing key as its properties; `html` prints one page with the same. Both are made from the signed
records alone, so they say what the records say; `json` is the envelope `--json` prints, and with `junit` or `html` the
progress goes to standard error.
- **`--independent`** beside a kind (`wai conform <kind> --suite <s> --impl <i> --independent`) runs the corpus's
independent verifier, its second implementation written from the extension's text, on the same cases: the claim, JWP,
key-discovery and conformance-result suites case by case through each verifier's `verdicts` mode, the HTTP origin
suite over its whole corpus. Its outcome is a second signed record beside the implementation's
(`<suite>.independent.wai-conform.json`), named `independent verifier`, so a report tells its test suite from the
implementation's. The run passes only when both conform and the two agree: every case on which they came to different
outcomes, or which both passed with different answers (a case the corpus states only the start of, such as an accepted
claim whose link hash it does not pin), is named and fails the run. The score and packaging policy suites have none
here, and say so; a test holds every suite to naming its corpus's verifier where the table has one.
- **`wai conform --independent`** alone runs every corpus's independent verifier (or those `--suite` names, by area,
`trust`, `media`, `delivery` or `live`, or by name), each in `wai/` with the command line its README gives, from one
table in the `wai` binary that `wai conform --list` prints; with `--out <dir>`, a signed record a corpus, which
`wai verify <dir>` checks. A verifier that fails shows the end of its output, and the Python package it needs, where
that is missing. `--python` names the interpreter (`python3`).
- **The corpus union is one command.** `wai-conformance.yml` ran each corpus's Python verifier in a step of its own, a
list kept by hand; it now runs them all as `wai conform --independent --format junit`, uploads the records and the
report, and checks the records with `wai verify`. The steps keep each corpus's Rust side. The table's tests require
every `*_verify.py` in the tree to be in it or to say where it runs instead (the opus deep redundancy and header probe
verifiers need fetched inputs and run in their steps, and `pack_verify.py` in `wai-pack.yml`), each run in a step's
command of the workflow it names, and `wai-conformance.yml` to run none of the table's by hand. A record's corpus digest
is of the files the run reads: a run given one vectors file digests that file (each of the OER corpus's four runs, and
the claims runs over the bearer, usage and carousel corpora), and one given a directory, every file in it. The union
gains the pin corpus's verifier, which ran on the macOS leg alone, and the OER corpus's, `oer2_verify.py`, over its
four vector files and its unreadable documents, which no workflow ran.
- `wai mcp`'s `conformance_list` and `conformance_run` read the same table: each corpus with its command line, and a
corpus run by its directory or by its verifier's name. A directory the claim verifier also runs over
(`live-conformance/bearers`, `usage`, `carousel`) runs its corpus's own verifier, as it did before the claims runs were
in the table; one with several of its own (`oer-conformance`) names them.

14
wai/changes/2026-11-04-wai-doctor-which-wai-runs-you.md Normal file
View file

@ -0,0 +1,14 @@
### `wai doctor`: which `wai` runs when you type it, and what the quickstart needs
- **`wai doctor`** (operation `toolchain.doctor`, preview; in the agent door as `toolchain_doctor`) checks, in order:
which program runs when a person types `wai` (another program of that name first on `PATH` fails, and the one the
crates.io package `wai-cli` installs, which `cargo`'s own list of what it installed names, is named as that, with
`cargo uninstall wai-cli`); this install, how it was installed, and the relay, the room service and the room client's
files `wai dev` runs, found as `wai dev` finds them; whether loopback can be listened on and the ports `wai dev` and
`wai http origin` take by default are free; the browsers the system has registered to open web pages; and where a
credential is kept (the keychain, the desktop secret service, or a file only its owner can read). Exit 1 when a check
fails; warnings do not. A package manager's shim first on `PATH`, as WAI's bucket installs, is the program its `.shim`
file names; and a release from before the catalogue (0.3.0) is another copy of WAI's command line, not another
program. The catalogue's verbs gain `doctor`, which the toolchain specification's operation table names; the check of
every door's names (`scripts/check-wai-ops-doors.py`) now reads the verbs and their near-synonyms from the catalogue
itself, so it holds `doctor`, `upscale` and `tonemap` too.

5
wai/quickstart/README.md
View file

@ -26,8 +26,9 @@ wai dev # the relay, the room service and yo
wai verify ../wai-hello # the receipts hello wrote, checked offline
```
`wai create --list` names every template. `wai dev --help` gives its options. Nothing here
needs an account, and nothing is sent off your machine.
`wai create --list` names every template. `wai dev --help` gives its options. `wai doctor` says
which `wai` runs when you type it and whether `wai dev` finds what it runs. Nothing here needs an
account, and nothing is sent off your machine.
Keep `bin/` and `share/` side by side: `wai dev` finds the relay, the room service, the room
client and Studio beside itself. Homebrew and Scoop install `wai-quantum` as a package of its own,

132
wai/quickstart/test/clean-machine.ps1 Normal file
View file

@ -0,0 +1,132 @@
# The quickstart as a new user runs it on Windows, from the archive alone, timed step by step:
#
# pwsh -File wai/quickstart/test/clean-machine.ps1 <archive.zip>
#
# The twin of clean-machine.sh, step for step. A machine with nothing of WAI on it: a fresh
# profile (USERPROFILE, HOME, APPDATA, LOCALAPPDATA, TEMP and TMP) and working directory, a PATH
# of the system's own directories, no cargo and no repository. The steps are the quickstart's:
# 1. check the archive against its .sha256 beside it, and unpack it;
# 2. the archive holds what `wai dev` and the quickstart name: bin/ with wai, wai-relay,
# wai-room, wai-ingest, wai-remote-host, wai-pack and wai-quantum, each of which runs, and
# share/wai/web/ and share/wai/studio/ (Studio's page and its core);
# 3. wai hello; 4. wai verify; 5. wai create my-room --template video-room;
# 6. wai dev --check in the project: the relay, the room service, the page and WAI Studio at
# /_wai/ on one origin, each checked through it.
# It prints each step's time, the total, and a JSON line; it exits non-zero when a step fails,
# naming it. Downloading the archive is not timed: the archive is given. It needs PowerShell 7
# and runs on any system it runs on (a .tar.gz archive elsewhere than Windows), so its steps can
# be checked off Windows too.
param([string]$Archive)
$ErrorActionPreference = 'Stop'
if (-not $Archive) {
Get-Content $PSCommandPath | Select-Object -First 18 | ForEach-Object { $_ -replace '^# ?', '' }
exit 2
}
$Archive = (Resolve-Path $Archive).Path
# The system's temporary directory, read before TEMP, TMP and TMPDIR point into the run's own.
$SystemTmp = [IO.Path]::GetTempPath()
$Root = Join-Path $SystemTmp ("wai-clean-" + [guid]::NewGuid().ToString('N'))
$HomeDir = Join-Path $Root 'home'
$Tmp = Join-Path $Root 'tmp'
$Work = Join-Path $Root 'work'
New-Item -ItemType Directory -Force -Path $HomeDir, $Tmp, $Work | Out-Null
$env:HOME = $HomeDir
$env:USERPROFILE = $HomeDir
$env:APPDATA = Join-Path $HomeDir 'AppData\Roaming'
$env:LOCALAPPDATA = Join-Path $HomeDir 'AppData\Local'
$env:TEMP = $Tmp
$env:TMP = $Tmp
$env:TMPDIR = $Tmp
New-Item -ItemType Directory -Force -Path $env:APPDATA, $env:LOCALAPPDATA | Out-Null
$Sep = [IO.Path]::PathSeparator
if ($IsWindows) {
$System = @("$env:SystemRoot\System32", $env:SystemRoot, "$env:SystemRoot\System32\Wbem", "$env:SystemRoot\System32\WindowsPowerShell\v1.0") -join $Sep
} else {
$System = '/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin'
}
$env:PATH = $System
Set-Location $Work
$T0 = [Diagnostics.Stopwatch]::StartNew()
$Steps = [ordered]@{}
$Failed = ''
# Run one step, its output to <name>.log: its time, or FAILED and the log's tail.
function Step([string]$Name, [scriptblock]$Do) {
$log = Join-Path $Root "$Name.log"
$w = [Diagnostics.Stopwatch]::StartNew()
$ok = $true
try {
$global:LASTEXITCODE = 0
& $Do *> $log
if ($LASTEXITCODE -ne 0) { $ok = $false }
} catch {
$_ | Out-File -Append $log
$ok = $false
}
if ($ok) {
$script:Steps[$Name] = [math]::Round($w.Elapsed.TotalSeconds, 2).ToString([Globalization.CultureInfo]::InvariantCulture)
} else {
$script:Steps[$Name] = 'FAILED'
$script:Failed = $Name
Write-Output "FAIL ${Name}:"
if (Test-Path $log) { Get-Content $log -Tail 15 }
}
}
Step 'unpack' {
$want = ((Get-Content "$Archive.sha256" -Raw).Trim() -split '\s+')[0].ToLower()
$got = (Get-FileHash $Archive -Algorithm SHA256).Hash.ToLower()
if ($got -ne $want) { throw "SHA-256 $got, not $want" }
if ($Archive.EndsWith('.zip')) {
Expand-Archive -Path $Archive -DestinationPath $Work
} else {
tar -xzf $Archive -C $Work
}
}
$Bin = Get-ChildItem -Path $Work -Directory -Filter 'wai-*' | Select-Object -First 1 | ForEach-Object { Join-Path $_.FullName 'bin' }
if ($Bin) { $env:PATH = "$Bin$Sep$env:PATH" }
if (-not $Failed) {
Step 'contents' {
$exe = if ($IsWindows) { '.exe' } else { '' }
$missing = @()
foreach ($b in 'wai', 'wai-relay', 'wai-room', 'wai-ingest', 'wai-remote-host', 'wai-pack', 'wai-quantum') {
if (-not (Test-Path -PathType Leaf (Join-Path $Bin "$b$exe"))) { $missing += "bin/$b$exe" }
}
foreach ($f in 'share/wai/web/room/index.html', 'share/wai/studio/index.html', 'share/wai/studio/pkg/wai_studio_core_bg.wasm') {
if (-not (Test-Path -PathType Leaf (Join-Path (Split-Path $Bin -Parent) $f))) { $missing += $f }
}
if ($missing) { throw "the archive lacks: $($missing -join ' ')" }
wai-quantum --version
if ($LASTEXITCODE -ne 0) { throw "wai-quantum --version exits $LASTEXITCODE" }
# wai-ingest without a configuration says how it is run, and exits 2.
$err = Join-Path $Root 'ingest.err'
$p = Start-Process -FilePath (Join-Path $Bin "wai-ingest$exe") -NoNewWindow -Wait -PassThru -RedirectStandardOutput (Join-Path $Root 'ingest.out') -RedirectStandardError $err
if ($p.ExitCode -ne 2 -or -not (Select-String -Path $err -SimpleMatch -Quiet 'usage: wai-ingest')) {
throw "wai-ingest without a configuration exits $($p.ExitCode), not 2 with its usage"
}
}
}
if (-not $Failed) { Step 'hello' { wai hello } }
if (-not $Failed) { Step 'verify' { wai verify wai-hello } }
if (-not $Failed) { Step 'create' { wai create my-room --template video-room } }
# Every piece held, Studio among them: the archive's share/wai/studio, found beside wai.
if (-not $Failed) { Step 'dev-check' { Push-Location my-room; try { wai dev --check --port 0 | Tee-Object -Variable out; if ($LASTEXITCODE -eq 0 -and -not ($out | Select-String -SimpleMatch -Quiet 'ok WAI Studio at /_wai/')) { throw 'wai dev --check did not hold WAI Studio at /_wai/' } } finally { Pop-Location } } }
$Total = [math]::Round($T0.Elapsed.TotalSeconds, 2).ToString([Globalization.CultureInfo]::InvariantCulture)
$Os = [Runtime.InteropServices.RuntimeInformation]::OSDescription
$Arch = [Runtime.InteropServices.RuntimeInformation]::OSArchitecture
Write-Output "the quickstart from $(Split-Path $Archive -Leaf) on $Os $Arch, a fresh profile and directory:"
foreach ($k in $Steps.Keys) {
if ($Steps[$k] -eq 'FAILED') { Write-Output (" {0,-10} FAILED" -f $k) } else { Write-Output (" {0,-10} {1} s" -f $k, $Steps[$k]) }
}
$devLog = Join-Path $Root 'dev-check.log'
if (Test-Path $devLog) { Select-String -Path $devLog -Pattern 'checks held' | ForEach-Object { " $($_.Line)" } }
Write-Output (" total $Total s" + $(if ($Failed) { " (stopped at $Failed)" } else { '' }))
$stepsOut = [ordered]@{}
foreach ($k in $Steps.Keys) { $stepsOut[$k] = $Steps[$k] }
[ordered]@{ archive = (Split-Path $Archive -Leaf); total_s = [double]$Total; steps = $stepsOut; ok = (-not $Failed) } | ConvertTo-Json -Compress
Set-Location $SystemTmp
Remove-Item -Recurse -Force $Root -ErrorAction SilentlyContinue
if ($Failed) { exit 1 }
exit 0

3
wai/quickstart/test/clean-machine.sh
View file

@ -15,7 +15,8 @@
# 7. with --browser (BROWSER_BIN and PLAYWRIGHT_HOST set): the project's page joined in two
# browser pages, each decoding the other (the video-room example's test).
# It prints each step's time, the total, and a JSON line; it exits non-zero when a step fails,
# naming it. Downloading the archive is not timed: the archive is given.
# naming it. Downloading the archive is not timed: the archive is given. Its PowerShell twin,
# clean-machine.ps1, runs the same steps from the Windows archive.
set -uo pipefail
[ $# -ge 1 ] || { sed -n '2,19p' "$0"; exit 2; }
ARCHIVE=$(cd "$(dirname "$1")" && pwd)/$(basename "$1")

32
wai/wai-rs/build.rs
View file

@ -13,9 +13,13 @@
#[path = "src/build_info.rs"]
mod build_info;
#[path = "src/bin/wai_cli/dx/independent_table.rs"]
mod independent_table;
fn main() {
generate_c_header();
record_build();
independent_verifiers();
// Only the `codecs` feature compiles flac-bound — and pkg-config's
// probe() emits a `cargo:rustc-link-lib=flac` of its own, so running
// it unconditionally makes every consumer (e.g. the wasm32 world
@ -51,6 +55,34 @@ fn record_build() {
}
}
/// The `wai` binary's table of independent verifiers (`dx/independent.rs`), written to
/// `$OUT_DIR/independent_verifiers.rs` from the `# independent:` lines of the conformance
/// workflow's step files (`.github/wai-conformance/verifiers/*.sh`), which hold no list a change
/// appends to. A build outside the repository, which has no step files, gets an empty table
/// and the reason (`TABLE_ABSENT`), so `wai conform --independent` refuses to run rather than
/// pass on nothing; so does a build whose step files cannot be read, with the reason.
fn independent_verifiers() {
let manifest = std::path::PathBuf::from(std::env::var("CARGO_MANIFEST_DIR").unwrap());
let repo = manifest.join("../..");
let steps = repo.join(independent_table::STEPS);
println!("cargo:rerun-if-changed={}", steps.display());
let text = match independent_table::entries(&repo) {
Ok(Some(e)) => independent_table::render(&e),
Ok(None) => independent_table::render_absent(&format!(
"this wai was built without {}/, the conformance workflow's step files whose `# independent:` lines are the table of independent verifiers, so it has none to run; build it from a checkout of the repository",
independent_table::STEPS
)),
Err(why) => {
println!("cargo:warning=the table of independent verifiers: {why}");
independent_table::render_absent(&format!("this wai was built from step files it could not read: {why}"))
}
};
let out = std::path::PathBuf::from(std::env::var("OUT_DIR").unwrap()).join("independent_verifiers.rs");
if std::fs::read_to_string(&out).ok().as_deref() != Some(text.as_str()) {
std::fs::write(&out, text).expect("write the table of independent verifiers");
}
}
/// Regenerate `include/wai.h` from the FFI surface so SDK consumers can
/// always grab a typed header that matches the shipped binary. Only
/// regenerates when source files actually change (cargo handles this

657
wai/wai-rs/src/bin/wai_cli/dx/conform/mod.rs
View file

@ -9,6 +9,21 @@
//! implementation it ran (the command and the SHA-256 of every file it names, or the URL),
//! each case's outcome, and the standard's version.
//!
//! `--independent` runs the corpus's independent verifier (`dx::independent`: its second
//! implementation, written from the text and sharing no code with wai-rs) beside the
//! implementation, held to the same cases, its outcome a second record; the run passes only
//! when both conform and the two agree on every case. A case on which they disagree is named:
//! one passes it and the other does not, or both pass it with different answers (a case the
//! corpus states only the start of, such as an accepted claim whose link hash it does not pin).
//! Without a kind,
//! `wai conform --independent [--suite <area|name>[,…]]` runs every independent verifier (or
//! those named) over its corpus: the corpus union CI runs as this one command, a signed record
//! a corpus with `--out <dir>`.
//!
//! `--format` says what is printed: `text` (the default), `json` (the envelope, as `--json`),
//! `junit` (JUnit XML, a test suite a record) or `html` (one page); the last two are made from
//! the signed records alone (`report`), and progress then goes to standard error.
//!
//! Exit status: 0 when every case passed, 1 when one did not, 2 for a usage error, 4 when
//! the record would replace a file and `--yes` was not given, 5 when the suite, corpus
//! or program is not found, 6 when the implementation's URL does not answer, 7 otherwise.
@ -16,9 +31,11 @@
mod adapter;
mod http;
pub mod record;
mod report;
pub mod suites;
use super::envelope::{self, EXIT_FAILED, EXIT_INTERNAL, EXIT_NOT_FOUND, EXIT_UNAVAILABLE, OpError};
use super::independent::{self, Verifier};
use serde_json::{Map, Value, json};
use std::collections::HashMap;
use std::path::{Path, PathBuf};
@ -27,10 +44,48 @@ use suites::{Case, Expect, Input, SUITES, Suite};
pub const OP: &str = "conformance.run";
const USAGE: &str = "wai conform <kind> --suite <suite> --impl <command or URL> [--key <file>] [--out <file>] [--yes]\n\
[--root <wai-dir>] [--timeout <seconds>] [--name <text>] [--version <text>] [--dry-run] [--all] [--json|--text]\n\
const USAGE: &str = "wai conform <kind> --suite <suite> --impl <command or URL> [--independent] [--key <file>] [--out <file>] [--yes]\n\
[--root <wai-dir>] [--timeout <seconds>] [--name <text>] [--version <text>] [--python <program>] [--dry-run] [--all]\n\
[--format text|json|junit|html]\n\
wai conform --independent [--suite <area|name>[,...]] [--out <dir>] [--key <file>] [--root <wai-dir>] [--timeout <seconds>]\n\
[--python <program>] [--format text|json|junit|html] [--dry-run] [--yes]\n\
wai conform --list";
/// What a run prints on standard output.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Format {
Text,
/// The JSON envelope (`--json`).
Json,
/// JUnit XML, from the records.
Junit,
/// One HTML page, from the records.
Html,
}
/// The output `--format` (or `--json`, or `--text`) asks for; two that disagree are a usage error.
pub fn format_of(args: &[String]) -> Result<Format, OpError> {
let json = envelope::json_wanted(args);
let named = match envelope::value(args, "--format", USAGE)? {
None => return Ok(if json { Format::Json } else { Format::Text }),
Some("text") => Format::Text,
Some("json") => Format::Json,
Some("junit") => Format::Junit,
Some("html") => Format::Html,
Some(x) => return Err(OpError::usage(format!("--format {x}: text, json, junit or html"), USAGE)),
};
if json && named != Format::Json {
return Err(OpError::usage("--json and --format name two outputs; give one", USAGE));
}
Ok(named)
}
/// The seconds an independent verifier may take over its whole corpus, unless `--timeout` says.
const INDEPENDENT_TIMEOUT: u64 = 1800;
/// What an independent run's one case expects.
const WHOLE_CORPUS: &str = "exit 0: every case reaches the outcome the corpus states";
/// How the implementation is reached.
enum Impl {
Command(String),
@ -365,18 +420,28 @@ impl Run<'_> {
}
}
fn list(json_mode: bool, root: Option<&Path>) {
fn list(json_mode: bool, root: Option<&Path>, python: &str) {
let rows: Vec<Value> = SUITES
.iter()
.map(|s| {
let n = root.and_then(|r| (s.load)(&r.join(s.dir)).ok()).map(|l| l.cases.len());
let adapters: Vec<&str> = [s.command.then_some("command"), s.http.then_some("http")].into_iter().flatten().collect();
json!({"kind": s.kind, "suite": s.name, "corpus": s.dir, "cases": n, "adapters": adapters, "about": s.about})
json!({"kind": s.kind, "suite": s.name, "corpus": s.dir, "cases": n, "adapters": adapters, "about": s.about, "independent": s.independent})
})
.collect();
let ind: Vec<Value> = independent::VERIFIERS
.iter()
.map(|v| json!({"name": v.name, "area": v.area, "corpus": v.dir, "command": independent::command_line(v, python, false), "verdicts": v.verdicts.is_some(), "needs": v.needs, "step_file": v.file}))
.collect();
let elsewhere: Vec<Value> = independent::NOT_HERE.iter().map(|(s, why)| json!({"verifier": s, "why": why})).collect();
if json_mode {
let mut m = Map::new();
m.insert("suites".into(), Value::Array(rows));
m.insert("independent".into(), Value::Array(ind));
m.insert("independent_elsewhere".into(), Value::Array(elsewhere));
if let Some(why) = independent::TABLE_ABSENT {
m.insert("independent_absent".into(), json!(why));
}
println!("{}", envelope::envelope("conformance.list", true, m));
return;
}
@ -391,28 +456,228 @@ fn list(json_mode: bool, root: Option<&Path>) {
r["about"].as_str().unwrap_or("")
);
}
println!("\nindependent verifiers, run in wai/ by `wai conform --independent [--suite <area|name>,...]`:");
if let Some(why) = independent::TABLE_ABSENT {
println!(" none: {why}");
}
for r in ind {
println!("{:<9} {:<22} {}", r["area"].as_str().unwrap_or(""), r["name"].as_str().unwrap_or(""), r["command"].as_str().unwrap_or(""));
}
println!("\nrun elsewhere, not by --independent:");
for r in elsewhere {
println!(" {}: {}", r["verifier"].as_str().unwrap_or(""), r["why"].as_str().unwrap_or(""));
}
}
pub fn main(args: &[String]) -> Result<bool, String> {
let json_mode = envelope::json_wanted(args);
match run(args, json_mode) {
let json_mode = matches!(format_of(args), Ok(Format::Json)) || envelope::json_wanted(args);
match run(args) {
Ok(true) => Ok(true),
Ok(false) => envelope::exit(EXIT_FAILED),
Err(e) => envelope::fail(OP, json_mode, &e),
}
}
fn run(args: &[String], json_mode: bool) -> Result<bool, OpError> {
let flags = ["--suite", "--impl", "--key", "--out", "--root", "--timeout", "--name", "--version", "--dry-run", "--all", "--json", "--text", "--list"];
let with_value = ["--suite", "--impl", "--key", "--out", "--root", "--timeout", "--name", "--version"];
/// The parts of a record but its outcomes and signature.
struct Head<'a> {
kind: &'a str,
suite: &'a str,
corpus_dir: &'a str,
corpus_files: usize,
digest: String,
adapter: &'a str,
imp: String,
name: Option<&'a str>,
version: Option<&'a str>,
files: Vec<Value>,
started: String,
}
/// A record (conformance-results §2) of `outcomes`, signed with `key` (§3).
fn record_of(root: &Path, h: Head, outcomes: &[Outcome], key: &(ed25519_dalek::SigningKey, &str)) -> Result<Map<String, Value>, OpError> {
let passed = outcomes.iter().filter(|o| o.pass).count();
let mut rec = Map::new();
rec.insert("wai".into(), json!(record::FORMAT));
rec.insert("kind".into(), json!(h.kind));
rec.insert("suite".into(), json!(h.suite));
rec.insert("standard".into(), json!(standard_version(root)));
rec.insert("toolchain".into(), json!({"name": "wai", "version": env!("CARGO_PKG_VERSION")}));
rec.insert("corpus".into(), json!({"dir": h.corpus_dir, "files": h.corpus_files, "digest": h.digest}));
rec.insert("implementation".into(), json!({"adapter": h.adapter, "impl": h.imp, "name": h.name, "version": h.version, "files": h.files}));
rec.insert("started_at".into(), json!(h.started));
rec.insert("cases".into(), Value::Array(outcomes.iter().map(|o| json!({"name": o.name, "pass": o.pass, "expected": o.expected, "got": o.got})).collect()));
rec.insert("summary".into(), json!({"cases": outcomes.len(), "passed": passed, "failed": outcomes.len() - passed}));
rec.insert("conformant".into(), json!(!outcomes.is_empty() && passed == outcomes.len()));
rec.insert("key".into(), json!(key.1));
record::sign(rec, &key.0).map_err(|e| OpError::new(EXIT_INTERNAL, "internal", e, ""))
}
fn write_record(path: &Path, rec: &Map<String, Value>) -> Result<(), OpError> {
let text = serde_json::to_string_pretty(&Value::Object(rec.clone())).map_err(|e| OpError::new(EXIT_INTERNAL, "internal", e.to_string(), ""))?;
crate::out::write(path, format!("{text}\n")).map_err(|e| OpError::new(EXIT_INTERNAL, "internal", format!("{}: {e}", path.display()), ""))
}
/// The key a run signs with: `--key`'s file, or one made for the run.
fn run_key(given: Option<&str>) -> Result<(ed25519_dalek::SigningKey, &'static str), OpError> {
match given {
Some(p) => Ok((record::key_from_file(Path::new(p)).map_err(|e| OpError::new(EXIT_NOT_FOUND, "key-not-read", e, "--key names a file holding a 32-byte Ed25519 seed in hexadecimal"))?, "supplied")),
None => Ok((record::ephemeral_key().map_err(|e| OpError::new(EXIT_INTERNAL, "internal", e, ""))?, "ephemeral")),
}
}
/// The files an independent verifier's command line names, with their SHA-256 (§2.2): the
/// interpreter as found on `PATH`, the script, and every argument that is a file.
fn verifier_files(v: &Verifier, root: &Path, python: &str, verdicts: bool) -> Vec<Value> {
let mut out = Vec::new();
let python_path = if python.contains('/') || python.contains('\\') {
Some(PathBuf::from(python))
} else {
std::env::var_os("PATH").and_then(|p| std::env::split_paths(&p).map(|d| d.join(python)).find(|p| p.is_file()))
};
if let Some(p) = python_path
&& let Ok(b) = std::fs::read(&p)
{
out.push(json!({"path": p.to_string_lossy(), "sha256": adapter::sha256_hex(&b)}));
}
for w in independent::argv(v, verdicts) {
if let Ok(b) = std::fs::read(root.join(w))
&& root.join(w).is_file()
{
out.push(json!({"path": w, "sha256": adapter::sha256_hex(&b)}));
}
}
out
}
/// A failed independent verifier's answer: how it ended, and a package it needs that is missing.
fn independent_got(v: &Verifier, ran: &independent::Ran, python: &str) -> String {
let mut got = ran.verdict();
if !ran.passed() {
let missing = independent::missing_needs(v, python);
if !missing.is_empty() {
got.push_str(&format!(" (it needs the Python package {}: {python} -m pip install {})", missing.join(", "), missing.join(" ")));
}
}
cut(&got)
}
/// The corpus's independent verifier, run beside an implementation's run of `suite`.
struct Beside {
verifier: &'static Verifier,
command: String,
record: Map<String, Value>,
conformant: bool,
/// Cases where the implementation and the independent verifier disagree ([`disagreements`]).
disagreements: Vec<Value>,
}
/// What an independent verifier's record beside an implementation's names it, so the two are
/// told apart in a report (`implementation.name`).
const INDEPENDENT_NAME: &str = "independent verifier";
/// The cases, held case by case, on which the implementation (`imp`) and the independent
/// verifier (`ind`) disagree: one passes and the other does not, or both pass with different
/// answers, which a case the corpus states only the start of allows.
fn disagreements(imp: &[Outcome], ind: &[Outcome]) -> Vec<Value> {
imp.iter()
.zip(ind)
.filter(|(a, b)| a.pass != b.pass || (a.pass && a.got != b.got))
.map(|(a, b)| json!({"name": a.name, "expected": a.expected, "implementation": a.got, "independent": b.got}))
.collect()
}
#[allow(clippy::too_many_arguments)]
fn beside(root: &Path, kind: &str, suite: &Suite, loaded: &suites::Loaded, imp: &[Outcome], digest: &str, python: &str, timeout: Duration, key: &(ed25519_dalek::SigningKey, &str)) -> Result<Option<Beside>, OpError> {
let Some(name) = suite.independent else { return Ok(None) };
let v = independent::named(name).ok_or_else(|| OpError::internal(format!("{kind} {}: the independent verifier {name} is not in the table", suite.name)))?;
let verdicts = v.verdicts.is_some() && loaded.corpus_file.is_some();
let command = independent::command_line(v, python, verdicts);
let started = now_rfc3339();
eprintln!("wai conform: the independent verifier, {command}");
let ran = independent::run(v, root, python, verdicts, timeout).map_err(|e| OpError::new(EXIT_NOT_FOUND, "program-not-found", e, "install Python 3, or give --python <program>"))?;
let (outcomes, head) = if verdicts {
let mut got: HashMap<String, String> = HashMap::new();
for l in ran.stdout.lines() {
if let Some((n, g)) = l.split_once('\t') {
got.entry(n.to_string()).or_insert_with(|| g.trim().to_string());
}
}
let missing = if ran.timed_out { "timed out".to_string() } else { format!("no verdict ({})", independent_got(v, &ran, python)) };
let o: Vec<Outcome> = loaded
.cases
.iter()
.map(|c| {
let g = got.get(&c.name).cloned().unwrap_or_else(|| missing.clone());
Outcome { name: c.name.clone(), pass: judge(&c.expect, &g, None), expected: cut(&expected_text(&c.expect)), got: cut(&g) }
})
.collect();
let head = Head {
kind,
suite: suite.name,
corpus_dir: suite.dir,
corpus_files: loaded.files.len(),
digest: digest.to_string(),
adapter: "command",
imp: command.clone(),
name: Some(INDEPENDENT_NAME),
version: None,
files: verifier_files(v, root, python, true),
started,
};
(o, head)
} else {
let files = independent::corpus_files(v, root);
let head = Head {
kind: "verifier",
suite: v.name,
corpus_dir: v.dir,
corpus_files: files.len(),
digest: corpus_digest(root, &files)?,
adapter: "command",
imp: command.clone(),
name: Some(INDEPENDENT_NAME),
version: None,
files: verifier_files(v, root, python, false),
started,
};
(vec![Outcome { name: v.dir.to_string(), pass: ran.passed(), expected: WHOLE_CORPUS.into(), got: independent_got(v, &ran, python) }], head)
};
let disagreements: Vec<Value> = if verdicts { disagreements(imp, &outcomes) } else { Vec::new() };
let record = record_of(root, head, &outcomes, key)?;
let conformant = record.get("conformant") == Some(&Value::Bool(true));
Ok(Some(Beside { verifier: v, command, record, conformant, disagreements }))
}
/// Where the independent verifier's record goes beside the implementation's record `out`.
fn beside_path(out: &Path) -> PathBuf {
let s = out.to_string_lossy();
PathBuf::from(match s.strip_suffix(record::SUFFIX) {
Some(stem) => format!("{stem}.independent{}", record::SUFFIX),
None => format!("{s}.independent{}", record::SUFFIX),
})
}
fn run(args: &[String]) -> Result<bool, OpError> {
let flags = [
"--suite", "--impl", "--key", "--out", "--root", "--timeout", "--name", "--version", "--dry-run", "--all", "--json", "--text", "--list", "--independent", "--format", "--python",
];
let with_value = ["--suite", "--impl", "--key", "--out", "--root", "--timeout", "--name", "--version", "--format", "--python"];
let pos = envelope::check_args(args, &flags, &with_value, 1, USAGE)?;
let v = |f: &str| envelope::value(args, f, USAGE);
let fmt = format_of(args)?;
let python = v("--python")?.unwrap_or("python3").to_string();
if args.iter().any(|a| a == "--list") {
let root = wai_root(v("--root")?).ok();
list(json_mode, root.as_deref());
list(fmt == Format::Json, root.as_deref(), &python);
return Ok(true);
}
let kind = *pos.first().ok_or_else(|| OpError::usage(format!("name a kind: {}", suites::KINDS.join(", ")), USAGE))?;
let with_independent = args.iter().any(|a| a == "--independent");
let Some(kind) = pos.first().copied() else {
if with_independent {
return run_independent(args, fmt, &python);
}
return Err(OpError::usage(format!("name a kind ({}), or --independent for the corpora's own verifiers", suites::KINDS.join(", ")), USAGE));
};
if !suites::KINDS.contains(&kind) {
return Err(OpError::new(EXIT_NOT_FOUND, "kind-not-found", format!("{kind}: no suite of that kind; the kinds are {}", suites::KINDS.join(", ")), "wai conform --list names every kind and suite"));
}
@ -439,7 +704,8 @@ fn run(args: &[String], json_mode: bool) -> Result<bool, OpError> {
}
Impl::Command(imp_text.clone())
};
let timeout = Duration::from_secs(v("--timeout")?.map(|t| t.parse::<u64>().map_err(|_| OpError::usage("--timeout takes whole seconds", USAGE))).transpose()?.unwrap_or(60));
let timeout_given = v("--timeout")?.map(|t| t.parse::<u64>().map_err(|_| OpError::usage("--timeout takes whole seconds", USAGE))).transpose()?;
let timeout = Duration::from_secs(timeout_given.unwrap_or(60));
let root = wai_root(v("--root")?)?;
let loaded = (suite.load)(&root.join(suite.dir)).map_err(|e| OpError::new(EXIT_NOT_FOUND, "corpus-not-read", e, "the corpus is incomplete; check out the standard again"))?;
if loaded.cases.is_empty() {
@ -450,9 +716,20 @@ fn run(args: &[String], json_mode: bool) -> Result<bool, OpError> {
return Err(OpError::usage(format!("{kind} {}: {{corpus}} is for a verifier suite; this suite gives its cases one at a time", suite.name), USAGE));
}
let out = PathBuf::from(v("--out")?.map(String::from).unwrap_or_else(|| format!("{}{}", suite.name, record::SUFFIX)));
if with_independent && let Some(why) = independent::TABLE_ABSENT {
return Err(OpError::new(EXIT_NOT_FOUND, "no-independent-table", why, NO_TABLE_HINT));
}
let independent_entry = suite.independent.and_then(independent::named);
let out_beside = (with_independent && independent_entry.is_some()).then(|| beside_path(&out));
if with_independent && independent_entry.is_some() && !independent::program_found(&python) {
return Err(OpError::new(EXIT_NOT_FOUND, "program-not-found", format!("{python}: not found, and --independent runs the corpus's verifier with it"), "install Python 3, or give --python <program>"));
}
if args.iter().any(|a| a == "--dry-run") {
// The record it would write, in the envelope's plan.
// The records it would write, in the envelope's plan.
crate::out::would_write(&out);
if let Some(b) = &out_beside {
crate::out::would_write(b);
}
let first = &loaded.cases[0];
let plan = match &imp {
Impl::Command(c) if whole => adapter::fill(c, &[("corpus", loaded.corpus_file.as_deref().unwrap_or(Path::new("")))]),
@ -462,30 +739,36 @@ fn run(args: &[String], json_mode: bool) -> Result<bool, OpError> {
}), ("out", Path::new("<out>"))]),
Impl::Http(u) => format!("{} {}", if matches!(first.expect, Expect::Object { .. }) { "GET, HEAD and OPTIONS below" } else { "POST each case to" }, u.join("")),
};
if json_mode {
let beside_plan = independent_entry.filter(|_| with_independent).map(|e| independent::command_line(e, &python, e.verdicts.is_some() && loaded.corpus_file.is_some()));
if fmt == Format::Json {
let mut m = Map::new();
m.insert("kind".into(), json!(kind));
m.insert("suite".into(), json!(suite.name));
m.insert("dry_run".into(), json!(true));
m.insert("cases".into(), json!(loaded.cases.iter().map(|c| c.name.clone()).collect::<Vec<_>>()));
m.insert("runs".into(), json!(plan));
m.insert("independent".into(), json!(beside_plan));
m.insert("record".into(), json!(out.display().to_string()));
println!("{}", envelope::envelope(OP, true, m));
} else {
println!("wai conform {kind} {}: {} cases from {}; would run: {plan}", suite.name, loaded.cases.len(), suite.dir);
if let Some(b) = &beside_plan {
println!("and beside it, in {}: {b}", root.display());
} else if with_independent {
println!("the {} corpus has no independent verifier to run beside it", suite.dir);
}
println!("dry run: nothing was run and nothing written; the record would be written to {}", out.display());
}
return Ok(true);
}
if out.exists() {
let replaced: Vec<&PathBuf> = std::iter::once(&out).chain(out_beside.as_ref()).filter(|p| p.exists()).collect();
if !replaced.is_empty() {
// `--yes` (or its deprecated spelling `--confirm`), or yes at a terminal; otherwise exit 4
// with the proposal and the confirming command.
envelope::confirm(&format!("{} exists, and the record would replace it", out.display()), vec![json!({"write": out.display().to_string(), "replaces": true})])?;
let names: Vec<String> = replaced.iter().map(|p| p.display().to_string()).collect();
envelope::confirm(&format!("{} exist{}, and the record would replace {}", names.join(" and "), if names.len() == 1 { "s" } else { "" }, if names.len() == 1 { "it" } else { "them" }), replaced.iter().map(|p| json!({"write": p.display().to_string(), "replaces": true})).collect())?;
}
let key = match v("--key")? {
Some(p) => (record::key_from_file(Path::new(p)).map_err(|e| OpError::new(EXIT_NOT_FOUND, "key-not-read", e, "--key names a file holding a 32-byte Ed25519 seed in hexadecimal"))?, "supplied"),
None => (record::ephemeral_key().map_err(|e| OpError::new(EXIT_INTERNAL, "internal", e, ""))?, "ephemeral"),
};
let key = run_key(v("--key")?)?;
let digest = corpus_digest(&root, &loaded.files)?;
let started = now_rfc3339();
let scratch = std::env::temp_dir().join(format!("wai-conform-{}-{}", std::process::id(), std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_nanos()).unwrap_or(0)));
@ -516,65 +799,293 @@ fn run(args: &[String], json_mode: bool) -> Result<bool, OpError> {
Impl::Command(c) => ("command", adapter::identity(c).into_iter().map(|(p, h)| json!({"path": p, "sha256": h})).collect::<Vec<_>>()),
Impl::Http(_) => ("http", Vec::new()),
};
let mut rec = Map::new();
rec.insert("wai".into(), json!(record::FORMAT));
rec.insert("kind".into(), json!(kind));
rec.insert("suite".into(), json!(suite.name));
rec.insert("standard".into(), json!(standard_version(&root)));
rec.insert("toolchain".into(), json!({"name": "wai", "version": env!("CARGO_PKG_VERSION")}));
rec.insert("corpus".into(), json!({"dir": suite.dir, "files": loaded.files.len(), "digest": digest}));
rec.insert("implementation".into(), json!({"adapter": adapter_name, "impl": imp_text, "name": v("--name")?, "version": v("--version")?, "files": files}));
rec.insert("started_at".into(), json!(started));
rec.insert("cases".into(), Value::Array(outcomes.iter().map(|o| json!({"name": o.name, "pass": o.pass, "expected": o.expected, "got": o.got})).collect()));
rec.insert("summary".into(), json!({"cases": outcomes.len(), "passed": passed, "failed": failed}));
rec.insert("conformant".into(), json!(conformant));
rec.insert("key".into(), json!(key.1));
let rec = record::sign(rec, &key.0).map_err(|e| OpError::new(EXIT_INTERNAL, "internal", e, ""))?;
let text = serde_json::to_string_pretty(&Value::Object(rec.clone())).map_err(|e| OpError::new(EXIT_INTERNAL, "internal", e.to_string(), ""))?;
crate::out::write(&out, format!("{text}\n")).map_err(|e| OpError::new(EXIT_INTERNAL, "internal", format!("{}: {e}", out.display()), ""))?;
let pubkey = rec.get("signer_pubkey").and_then(Value::as_str).unwrap_or("").to_string();
if json_mode {
let mut m = Map::new();
m.insert("kind".into(), json!(kind));
m.insert("suite".into(), json!(suite.name));
m.insert("cases".into(), json!(outcomes.len()));
m.insert("passed".into(), json!(passed));
m.insert("failed".into(), json!(failed));
m.insert("conformant".into(), json!(conformant));
m.insert("failures".into(), Value::Array(outcomes.iter().filter(|o| !o.pass).take(200).map(|o| json!({"name": o.name, "expected": o.expected, "got": o.got})).collect()));
m.insert("record".into(), json!(out.display().to_string()));
m.insert("corpus_digest".into(), json!(rec["corpus"]["digest"]));
m.insert("signer_pubkey".into(), json!(pubkey));
println!("{}", envelope::envelope(OP, conformant, m));
let head = Head { kind, suite: suite.name, corpus_dir: suite.dir, corpus_files: loaded.files.len(), digest: digest.clone(), adapter: adapter_name, imp: imp_text.clone(), name: v("--name")?, version: v("--version")?, files, started };
let rec = record_of(&root, head, &outcomes, &key)?;
write_record(&out, &rec)?;
let indep = if with_independent {
let t = Duration::from_secs(timeout_given.unwrap_or(INDEPENDENT_TIMEOUT));
let b = beside(&root, kind, suite, &loaded, &outcomes, &digest, &python, t, &key)?;
if let (Some(b), Some(p)) = (&b, &out_beside) {
write_record(p, &b.record)?;
}
b
} else {
let all = args.iter().any(|a| a == "--all");
let mut shown = 0;
for o in &outcomes {
if o.pass && !all {
continue;
None
};
let all_ok = conformant && indep.as_ref().is_none_or(|b| b.conformant && b.disagreements.is_empty());
let pubkey = rec.get("signer_pubkey").and_then(Value::as_str).unwrap_or("").to_string();
let mut records = vec![rec.clone()];
if let Some(b) = &indep {
records.push(b.record.clone());
}
let say = |line: String| if fmt == Format::Text { println!("{line}") } else { eprintln!("{line}") };
match fmt {
Format::Json => {
let mut m = Map::new();
m.insert("kind".into(), json!(kind));
m.insert("suite".into(), json!(suite.name));
m.insert("cases".into(), json!(outcomes.len()));
m.insert("passed".into(), json!(passed));
m.insert("failed".into(), json!(failed));
m.insert("conformant".into(), json!(conformant));
m.insert("failures".into(), Value::Array(outcomes.iter().filter(|o| !o.pass).take(200).map(|o| json!({"name": o.name, "expected": o.expected, "got": o.got})).collect()));
m.insert("record".into(), json!(out.display().to_string()));
m.insert("corpus_digest".into(), json!(rec["corpus"]["digest"]));
m.insert("signer_pubkey".into(), json!(pubkey));
if with_independent {
m.insert(
"independent".into(),
match &indep {
None => Value::Null,
Some(b) => json!({
"verifier": b.verifier.name,
"command": b.command,
"conformant": b.conformant,
"cases": b.record["summary"]["cases"],
"passed": b.record["summary"]["passed"],
"failed": b.record["summary"]["failed"],
"failures": b.record["cases"].as_array().map(|c| c.iter().filter(|x| x["pass"] != true).take(200).cloned().collect::<Vec<_>>()).unwrap_or_default(),
"disagreements": b.disagreements,
"record": out_beside.as_ref().map(|p| p.display().to_string()),
}),
},
);
}
if !o.pass {
shown += 1;
if shown > 40 && !all {
println!("{}", envelope::envelope(OP, all_ok, m));
}
_ => {
let all = args.iter().any(|a| a == "--all");
let mut shown = 0;
for o in &outcomes {
if o.pass && !all {
continue;
}
if !o.pass {
shown += 1;
if shown > 40 && !all {
continue;
}
}
say(format!("{} {}", if o.pass { "PASS" } else { "FAIL" }, o.name));
if !o.pass {
say(format!(" expected: {}", o.expected));
say(format!(" got: {}", o.got));
}
}
println!("{} {}", if o.pass { "PASS" } else { "FAIL" }, o.name);
if !o.pass {
println!(" expected: {}", o.expected);
println!(" got: {}", o.got);
if shown > 40 && !all {
say(format!("… and {} more failures (--all lists every case)", shown - 40));
}
say(format!("wai conform {kind} {}: {passed} of {} cases passed: {}", suite.name, outcomes.len(), if conformant { "conformant" } else { "not conformant" }));
say(format!("record: {} (signed by {}…, {} key); `wai verify {}` checks it", out.display(), pubkey.get(..16).unwrap_or(""), key.1, out.display()));
if with_independent {
match &indep {
None => say(format!("independent: the {} corpus has no independent verifier", suite.dir)),
Some(b) => {
let s = &b.record["summary"];
say(format!(
"independent: {}: {} of {} cases passed: {}",
b.command,
s["passed"],
s["cases"],
if b.conformant { "it reaches every outcome the corpus states" } else { "it does not reach every outcome the corpus states" }
));
for c in b.record["cases"].as_array().into_iter().flatten().filter(|c| c["pass"] != true).take(40) {
say(format!("FAIL {} (independent)", c["name"].as_str().unwrap_or("")));
say(format!(" expected: {}", c["expected"].as_str().unwrap_or("")));
say(format!(" got: {}", c["got"].as_str().unwrap_or("")));
}
if b.disagreements.is_empty() {
if b.verifier.verdicts.is_some() {
say("independent: it and the implementation agree on every case".into());
}
} else {
say(format!("independent: it and the implementation disagree on {} case{}:", b.disagreements.len(), if b.disagreements.len() == 1 { "" } else { "s" }));
for d in b.disagreements.iter().take(40) {
say(format!(" {}: the implementation answered {}; the independent verifier {}; the corpus states {}", d["name"].as_str().unwrap_or(""), d["implementation"], d["independent"], d["expected"]));
}
}
if let Some(p) = &out_beside {
say(format!("record: {} (the independent verifier's)", p.display()));
}
}
}
}
match fmt {
Format::Junit => print!("{}", report::junit(&records)),
Format::Html => print!("{}", report::html(&records)),
_ => {}
}
}
if shown > 40 && !all {
println!("… and {} more failures (--all lists every case)", shown - 40);
}
println!(
"wai conform {kind} {}: {passed} of {} cases passed: {}",
suite.name,
outcomes.len(),
if conformant { "conformant" } else { "not conformant" }
);
println!("record: {} (signed by {}…, {} key); `wai verify {}` checks it", out.display(), pubkey.get(..16).unwrap_or(""), key.1, out.display());
}
Ok(conformant)
Ok(all_ok)
}
/// What a build without the table of independent verifiers is told to do.
const NO_TABLE_HINT: &str = "build wai from a checkout of the repository: build.rs reads the table from .github/wai-conformance/verifiers/";
/// `wai conform --independent`: every independent verifier `--suite` selects (all of them
/// without it), each over its corpus, in `wai/`; a record each, written to `--out <dir>`.
fn run_independent(args: &[String], fmt: Format, python: &str) -> Result<bool, OpError> {
let v = |f: &str| envelope::value(args, f, USAGE);
for f in ["--impl", "--name", "--version"] {
if v(f)?.is_some() {
return Err(OpError::usage(format!("{f} names an implementation of a kind; --independent alone runs the corpora's own verifiers (give a kind to run both)"), USAGE));
}
}
let chosen = independent::select(v("--suite")?).map_err(|e| match independent::TABLE_ABSENT {
Some(_) => OpError::new(EXIT_NOT_FOUND, "no-independent-table", e, NO_TABLE_HINT),
None => OpError::new(EXIT_NOT_FOUND, "suite-not-found", e, "wai conform --list names every area and independent verifier"),
})?;
let timeout = Duration::from_secs(v("--timeout")?.map(|t| t.parse::<u64>().map_err(|_| OpError::usage("--timeout takes whole seconds", USAGE))).transpose()?.unwrap_or(INDEPENDENT_TIMEOUT));
let root = wai_root(v("--root")?)?;
let out_dir = v("--out")?.map(PathBuf::from);
if out_dir.as_ref().is_some_and(|d| d.exists() && !d.is_dir()) {
return Err(OpError::usage("--out with --independent alone names a directory, for a record a corpus", USAGE));
}
let record_path = |n: &str| out_dir.as_ref().map(|d| d.join(format!("{n}{}", record::SUFFIX)));
if args.iter().any(|a| a == "--dry-run") {
let commands: Vec<Value> = chosen.iter().map(|e| json!({"name": e.name, "area": e.area, "command": independent::command_line(e, python, false)})).collect();
if let Some(d) = &out_dir {
crate::out::create_dir_all(d).map_err(|e| OpError::internal(e.to_string()))?;
}
for e in &chosen {
if let Some(p) = record_path(e.name) {
crate::out::would_write(p);
}
}
if fmt == Format::Json {
let mut m = Map::new();
m.insert("dry_run".into(), json!(true));
m.insert("independent".into(), Value::Array(commands));
m.insert("root".into(), json!(root.display().to_string()));
println!("{}", envelope::envelope(OP, true, m));
} else {
println!("wai conform --independent: {} verifier{}, each run in {}; would run:", chosen.len(), if chosen.len() == 1 { "" } else { "s" }, root.display());
for c in &commands {
println!(" {}", c["command"].as_str().unwrap_or(""));
}
println!("dry run: nothing was run and nothing written{}", out_dir.as_ref().map(|d| format!("; the records would be written to {}", d.display())).unwrap_or_default());
}
return Ok(true);
}
if !independent::program_found(python) {
return Err(OpError::new(EXIT_NOT_FOUND, "program-not-found", format!("{python}: not found; the independent verifiers are Python programs"), "install Python 3, or give --python <program>"));
}
let replaced: Vec<PathBuf> = chosen.iter().filter_map(|e| record_path(e.name)).filter(|p| p.exists()).collect();
if !replaced.is_empty() {
envelope::confirm(
&format!("{} record{} in {} exist{}, and this run would replace {}", replaced.len(), if replaced.len() == 1 { "" } else { "s" }, out_dir.as_ref().map(|d| d.display().to_string()).unwrap_or_default(), if replaced.len() == 1 { "s" } else { "" }, if replaced.len() == 1 { "it" } else { "them" }),
replaced.iter().map(|p| json!({"write": p.display().to_string(), "replaces": true})).collect(),
)?;
}
let key = run_key(v("--key")?)?;
let say = |line: String| if fmt == Format::Text { println!("{line}") } else { eprintln!("{line}") };
say(format!("wai conform --independent: {} verifier{}, each run in {}", chosen.len(), if chosen.len() == 1 { "" } else { "s" }, root.display()));
let mut records = Vec::new();
let mut rows = Vec::new();
for e in &chosen {
let command = independent::command_line(e, python, false);
let started = now_rfc3339();
let ran = independent::run(e, &root, python, false, timeout).map_err(|x| OpError::new(EXIT_NOT_FOUND, "program-not-found", x, "install Python 3, or give --python <program>"))?;
let pass = ran.passed();
let got = independent_got(e, &ran, python);
if pass {
say(format!("PASS {:<22} {:>7.1} s {command}", e.name, ran.seconds));
} else {
say(format!("FAIL {:<22} {:>7.1} s {command}", e.name, ran.seconds));
say(format!(" {got}"));
for l in ran.tail(40).lines() {
say(format!(" | {l}"));
}
}
let files = independent::corpus_files(e, &root);
let head = Head {
kind: "verifier",
suite: e.name,
corpus_dir: e.dir,
corpus_files: files.len(),
digest: corpus_digest(&root, &files)?,
adapter: "command",
imp: command.clone(),
name: None,
version: None,
files: verifier_files(e, &root, python, false),
started,
};
let rec = record_of(&root, head, &[Outcome { name: e.dir.to_string(), pass, expected: WHOLE_CORPUS.into(), got: got.clone() }], &key)?;
rows.push(json!({"name": e.name, "area": e.area, "corpus": e.dir, "command": command, "pass": pass, "exit_code": ran.code, "timed_out": ran.timed_out, "seconds": (ran.seconds * 10.0).round() / 10.0, "got": got, "output_tail": if pass { Value::Null } else { json!(ran.tail(40)) }}));
records.push(rec);
}
if let Some(d) = &out_dir {
crate::out::create_dir_all(d).map_err(|e| OpError::internal(format!("{}: {e}", d.display())))?;
for (e, r) in chosen.iter().zip(&records) {
write_record(&d.join(format!("{}{}", e.name, record::SUFFIX)), r)?;
}
}
let passed = rows.iter().filter(|r| r["pass"] == true).count();
let failed = rows.len() - passed;
let pubkey = records.first().and_then(|r| r.get("signer_pubkey")).and_then(Value::as_str).unwrap_or("").to_string();
let summary = format!(
"wai conform --independent: {passed} of {} independent verifiers passed{}",
rows.len(),
if failed == 0 { String::new() } else { format!("; failed: {}", rows.iter().filter(|r| r["pass"] != true).filter_map(|r| r["name"].as_str()).collect::<Vec<_>>().join(", ")) }
);
match fmt {
Format::Json => {
let mut m = Map::new();
m.insert("independent".into(), Value::Array(rows));
m.insert("cases".into(), json!(passed + failed));
m.insert("passed".into(), json!(passed));
m.insert("failed".into(), json!(failed));
m.insert("conformant".into(), json!(failed == 0));
m.insert("records".into(), json!(out_dir.as_ref().map(|d| d.display().to_string())));
m.insert("signer_pubkey".into(), json!(pubkey));
println!("{}", envelope::envelope(OP, failed == 0, m));
}
_ => {
say(summary);
if let Some(d) = &out_dir {
say(format!("records: {} ({} of them, signed by {}…, {} key); `wai verify {}` checks them", d.display(), records.len(), pubkey.get(..16).unwrap_or(""), key.1, d.display()));
}
match fmt {
Format::Junit => print!("{}", report::junit(&records)),
Format::Html => print!("{}", report::html(&records)),
_ => {}
}
}
}
Ok(failed == 0)
}
#[cfg(test)]
mod tests {
use super::*;
fn o(name: &str, pass: bool, got: &str) -> Outcome {
Outcome { name: name.into(), pass, expected: "accept link_hash=…".into(), got: got.into() }
}
#[test]
fn two_answers_that_both_pass_but_differ_are_a_disagreement() {
let imp = [o("a", true, "accept link_hash=aa"), o("b", true, "accept link_hash=aa"), o("c", false, "reject"), o("d", true, "x"), o("e", false, "y")];
let ind = [o("a", true, "accept link_hash=aa"), o("b", true, "accept link_hash=bb"), o("c", true, "accept link_hash=cc"), o("d", false, "z"), o("e", false, "w")];
let d = disagreements(&imp, &ind);
let names: Vec<&str> = d.iter().map(|x| x["name"].as_str().unwrap()).collect();
// b: both pass a case the corpus states only the start of, with two link hashes.
assert_eq!(names, vec!["b", "c", "d"], "{d:#?}");
assert_eq!((d[0]["implementation"].as_str(), d[0]["independent"].as_str()), (Some("accept link_hash=aa"), Some("accept link_hash=bb")));
}
#[test]
fn each_oer_runs_record_digests_the_file_it_reads() {
let root = Path::new(env!("CARGO_MANIFEST_DIR")).parent().unwrap();
let digest = |n: &str| corpus_digest(root, &independent::corpus_files(independent::named(n).unwrap(), root)).unwrap();
let d: Vec<String> = ["oer", "oer-firmware", "oer-edge", "oer-job"].iter().map(|n| digest(n)).collect();
for i in 0..d.len() {
for j in 0..i {
assert_ne!(d[i], d[j], "two OER runs over different vectors files have different corpus digests");
}
}
assert_eq!(d[0], corpus_digest(root, &[root.join("oer-conformance/vectors.json")]).unwrap(), "the digest of the one file it reads");
}
}

264
wai/wai-rs/src/bin/wai_cli/dx/conform/report.rs Normal file
View file

@ -0,0 +1,264 @@
//! A run's result records (`conform-result/1`) written for a reader other than `wai verify`:
//! JUnit XML for a CI system's test report, and one HTML page for a person. Both are made from
//! the signed records alone, so they say what the records say: a suite a record, a test case a
//! case, a failure exactly where a case did not pass, with what the corpus states and what the
//! implementation answered. A record's corpus digest, implementation, standard and signing key
//! go with it, so a reader can find the record a report came from.
use serde_json::{Map, Value};
/// Text made safe for XML 1.0 character data and attribute values: the five markup
/// characters escaped, and every character XML 1.0 does not allow (C0 controls but tab, line
/// feed and carriage return) replaced by U+FFFD.
pub fn xml_escape(s: &str) -> String {
let mut out = String::with_capacity(s.len());
for c in s.chars() {
match c {
'&' => out.push_str("&amp;"),
'<' => out.push_str("&lt;"),
'>' => out.push_str("&gt;"),
'"' => out.push_str("&quot;"),
'\'' => out.push_str("&apos;"),
'\t' | '\n' | '\r' => out.push(c),
c if (c as u32) < 0x20 || c == '\u{FFFE}' || c == '\u{FFFF}' => out.push('\u{FFFD}'),
c => out.push(c),
}
}
out
}
/// Text made safe for HTML: the five markup characters escaped, control characters replaced.
pub fn html_escape(s: &str) -> String {
xml_escape(s).replace("&apos;", "&#39;")
}
fn s<'a>(r: &'a Map<String, Value>, k: &str) -> &'a str {
r.get(k).and_then(Value::as_str).unwrap_or("")
}
fn sub<'a>(r: &'a Map<String, Value>, k: &str, m: &str) -> &'a str {
r.get(k).and_then(|v| v.get(m)).and_then(Value::as_str).unwrap_or("")
}
fn cases(r: &Map<String, Value>) -> &[Value] {
r.get("cases").and_then(Value::as_array).map(Vec::as_slice).unwrap_or(&[])
}
fn failed(r: &Map<String, Value>) -> usize {
cases(r).iter().filter(|c| c["pass"] != Value::Bool(true)).count()
}
/// A record's name in a report: its kind and its suite, and the implementation's name where the
/// record gives one (an independent verifier's record beside an implementation's gives
/// `independent verifier`), so two records of one suite are told apart.
pub fn title(r: &Map<String, Value>) -> String {
match sub(r, "implementation", "name") {
"" => format!("{} {}", s(r, "kind"), s(r, "suite")),
n => format!("{} {} ({n})", s(r, "kind"), s(r, "suite")),
}
}
/// A record's JUnit class name: `<kind>.<suite>`, then the implementation's name where the record
/// gives one, as one more component (`verifier.signer.independent_verifier`): a character other
/// than a letter, a digit, `-` or `_` becomes `_`, so the name adds one component and no more.
pub fn classname(r: &Map<String, Value>) -> String {
let mut c = format!("{}.{}", s(r, "kind"), s(r, "suite"));
let n = sub(r, "implementation", "name");
if !n.is_empty() {
c.push('.');
c.extend(n.chars().map(|ch| if ch.is_alphanumeric() || ch == '-' || ch == '_' { ch } else { '_' }));
}
c
}
/// The JUnit XML report of `records`: one `testsuite` a record, one `testcase` a case, and a
/// `failure` on each case that did not pass.
pub fn junit(records: &[Map<String, Value>]) -> String {
let total: usize = records.iter().map(|r| cases(r).len()).sum();
let fails: usize = records.iter().map(failed).sum();
let mut x = String::from("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n");
x += &format!("<testsuites name=\"wai conform\" tests=\"{total}\" failures=\"{fails}\" errors=\"0\">\n");
for r in records {
let class = xml_escape(&classname(r));
x += &format!(
" <testsuite name=\"{}\" tests=\"{}\" failures=\"{}\" errors=\"0\" skipped=\"0\" timestamp=\"{}\">\n",
xml_escape(&title(r)),
cases(r).len(),
failed(r),
xml_escape(s(r, "started_at").trim_end_matches('Z'))
);
x += " <properties>\n";
let props = [
("standard", s(r, "standard").to_string()),
("toolchain", format!("{} {}", sub(r, "toolchain", "name"), sub(r, "toolchain", "version"))),
("corpus", sub(r, "corpus", "dir").to_string()),
("corpus_digest", sub(r, "corpus", "digest").to_string()),
("adapter", sub(r, "implementation", "adapter").to_string()),
("implementation", sub(r, "implementation", "impl").to_string()),
("conformant", r.get("conformant").map(Value::to_string).unwrap_or_default()),
("key", s(r, "key").to_string()),
("signer_pubkey", s(r, "signer_pubkey").to_string()),
];
for (k, v) in props {
x += &format!(" <property name=\"{k}\" value=\"{}\"/>\n", xml_escape(&v));
}
x += " </properties>\n";
for c in cases(r) {
let name = xml_escape(c["name"].as_str().unwrap_or(""));
if c["pass"] == Value::Bool(true) {
x += &format!(" <testcase classname=\"{class}\" name=\"{name}\"/>\n");
} else {
let (e, g) = (c["expected"].as_str().unwrap_or(""), c["got"].as_str().unwrap_or(""));
x += &format!(" <testcase classname=\"{class}\" name=\"{name}\">\n");
x += &format!(
" <failure message=\"{}\" type=\"not-as-the-corpus-states\">expected: {}\ngot: {}</failure>\n",
xml_escape(&format!("expected {e}; got {g}")),
xml_escape(e),
xml_escape(g)
);
x += " </testcase>\n";
}
}
x += " </testsuite>\n";
}
x += "</testsuites>\n";
x
}
/// The HTML report of `records`: one page, a section a record, its facts and every case, the
/// failed ones marked.
pub fn html(records: &[Map<String, Value>]) -> String {
let total: usize = records.iter().map(|r| cases(r).len()).sum();
let fails: usize = records.iter().map(failed).sum();
let conformant = records.iter().filter(|r| r.get("conformant") == Some(&Value::Bool(true))).count();
let mut h = String::from("<!doctype html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n<title>WAI conformance report</title>\n<style>\n");
h += ":root{--bg:#fff;--fg:#1b1b1b;--muted:#5d5d5d;--line:#d9d9d9;--pass:#1f6f3f;--fail:#a4262c;--fail-bg:#fdf0f0}\n";
h += "@media (prefers-color-scheme: dark){:root{--bg:#141414;--fg:#e8e8e8;--muted:#a3a3a3;--line:#333;--pass:#6fcf97;--fail:#ff8a8a;--fail-bg:#2a1717}}\n";
h += "body{margin:0 auto;max-width:72rem;padding:1.5rem 1rem;background:var(--bg);color:var(--fg);font:15px/1.5 system-ui,sans-serif}\n";
h += "h1{font-size:1.4rem;margin:0 0 .25rem}h2{font-size:1.1rem;margin:2rem 0 .5rem}p{margin:.25rem 0}\n";
h += "dl{display:grid;grid-template-columns:max-content 1fr;gap:.15rem 1rem;margin:.5rem 0;color:var(--muted)}dd{margin:0;overflow-wrap:anywhere;color:var(--fg)}\n";
h += "table{border-collapse:collapse;width:100%;font-size:.9rem}th,td{text-align:left;vertical-align:top;padding:.3rem .5rem;border-bottom:1px solid var(--line);overflow-wrap:anywhere}\n";
h += "code{font:.85rem ui-monospace,monospace}.pass{color:var(--pass)}.fail{color:var(--fail);font-weight:600}tr.fail td{background:var(--fail-bg)}\n";
h += "</style>\n</head>\n<body>\n<h1>WAI conformance report</h1>\n";
h += &format!(
"<p>{} of {} case{} passed in {} run{}; {} of {} conformant. Each section is one signed record, which <code>wai verify</code> checks.</p>\n",
total - fails,
total,
if total == 1 { "" } else { "s" },
records.len(),
if records.len() == 1 { "" } else { "s" },
conformant,
records.len()
);
for r in records {
let ok = r.get("conformant") == Some(&Value::Bool(true));
let n = cases(r).len();
h += &format!(
"<section>\n<h2>{} — <span class=\"{}\">{}</span></h2>\n<p>{} of {} case{} passed.</p>\n<dl>\n",
html_escape(&title(r)),
if ok { "pass" } else { "fail" },
if ok { "conformant" } else { "not conformant" },
n - failed(r),
n,
if n == 1 { "" } else { "s" }
);
let facts = [
("implementation", sub(r, "implementation", "impl").to_string()),
("corpus", format!("{} ({})", sub(r, "corpus", "dir"), sub(r, "corpus", "digest"))),
("standard", s(r, "standard").to_string()),
("toolchain", format!("{} {}", sub(r, "toolchain", "name"), sub(r, "toolchain", "version"))),
("started", s(r, "started_at").to_string()),
("signed by", format!("{} ({} key)", s(r, "signer_pubkey"), s(r, "key"))),
];
for (k, v) in facts {
h += &format!("<dt>{k}</dt><dd><code>{}</code></dd>\n", html_escape(&v));
}
h += "</dl>\n<table>\n<thead><tr><th>case</th><th>outcome</th><th>expected</th><th>got</th></tr></thead>\n<tbody>\n";
for c in cases(r) {
let pass = c["pass"] == Value::Bool(true);
h += &format!(
"<tr class=\"{}\"><td><code>{}</code></td><td class=\"{}\">{}</td><td><code>{}</code></td><td><code>{}</code></td></tr>\n",
if pass { "pass" } else { "fail" },
html_escape(c["name"].as_str().unwrap_or("")),
if pass { "pass" } else { "fail" },
if pass { "pass" } else { "FAIL" },
html_escape(c["expected"].as_str().unwrap_or("")),
html_escape(c["got"].as_str().unwrap_or(""))
);
}
h += "</tbody>\n</table>\n</section>\n";
}
h += "</body>\n</html>\n";
h
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
fn record(cases: Value, conformant: bool) -> Map<String, Value> {
json!({
"wai": "conform-result/1", "kind": "verifier", "suite": "claim", "standard": "v0.4.0",
"toolchain": {"name": "wai", "version": "0.4.0"},
"corpus": {"dir": "claim-conformance", "files": 1, "digest": "sha256:00"},
"implementation": {"adapter": "command", "impl": "v <x> & 'y'", "name": null, "version": null, "files": []},
"started_at": "2026-11-04T10:00:00Z", "cases": cases,
"summary": {}, "conformant": conformant, "key": "ephemeral", "signer_pubkey": "ab", "sig": "cd"
})
.as_object()
.unwrap()
.clone()
}
#[test]
fn markup_and_characters_xml_refuses_are_escaped() {
assert_eq!(xml_escape("a<b>&\"c'"), "a&lt;b&gt;&amp;&quot;c&apos;");
assert_eq!(xml_escape("x\u{0}y\u{1b}z\tw\n"), "x\u{FFFD}y\u{FFFD}z\tw\n");
assert_eq!(html_escape("'"), "&#39;");
}
#[test]
fn junit_has_a_failure_exactly_where_a_case_failed() {
let r = record(json!([{"name": "ok", "pass": true, "expected": "a", "got": "a"}, {"name": "<bad>", "pass": false, "expected": "accept", "got": "reject \u{0} & more"}]), false);
let x = junit(&[r.clone(), r]);
assert!(x.starts_with("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites name=\"wai conform\" tests=\"4\" failures=\"2\""), "{x}");
assert_eq!(x.matches("<testsuite ").count(), 2);
assert_eq!(x.matches("<testcase ").count(), 4);
assert_eq!(x.matches("<failure ").count(), 2);
assert!(x.contains("name=\"&lt;bad&gt;\""), "{x}");
assert!(x.contains("got: reject \u{FFFD} &amp; more</failure>"), "{x}");
assert!(x.contains("<property name=\"implementation\" value=\"v &lt;x&gt; &amp; &apos;y&apos;\"/>"), "{x}");
assert!(!x.contains('\u{0}'));
// Every element opened is closed: the counts balance.
for t in ["testsuites", "testsuite", "properties", "failure"] {
assert_eq!(x.matches(&format!("<{t} ")).count() + x.matches(&format!("<{t}>")).count(), x.matches(&format!("</{t}>")).count(), "{t}");
}
}
#[test]
fn a_record_with_an_implementations_name_is_told_apart() {
let plain = record(json!([{"name": "ok", "pass": true, "expected": "a", "got": "a"}]), true);
let mut named = plain.clone();
named["implementation"]["name"] = json!("independent verifier");
assert_eq!((title(&plain), classname(&plain)), ("verifier claim".to_string(), "verifier.claim".to_string()));
assert_eq!((title(&named), classname(&named)), ("verifier claim (independent verifier)".to_string(), "verifier.claim.independent_verifier".to_string()));
let x = junit(&[plain.clone(), named.clone()]);
assert!(x.contains("<testsuite name=\"verifier claim\" ") && x.contains("<testsuite name=\"verifier claim (independent verifier)\" "), "{x}");
assert!(x.contains("classname=\"verifier.claim\" ") && x.contains("classname=\"verifier.claim.independent_verifier\" "), "{x}");
let h = html(&[plain, named]);
assert!(h.contains("<h2>verifier claim (independent verifier) — "), "{h}");
}
#[test]
fn html_names_every_case_and_marks_the_failed_ones() {
let r = record(json!([{"name": "ok", "pass": true, "expected": "a", "got": "a"}, {"name": "<script>", "pass": false, "expected": "x", "got": "y"}]), false);
let h = html(&[r]);
assert!(h.starts_with("<!doctype html>") && h.contains("<title>WAI conformance report</title>"));
assert!(h.contains("1 of 2 cases passed in 1 run; 0 of 1 conformant"), "{h}");
assert_eq!(h.matches("<tr class=\"fail\">").count(), 1);
assert_eq!(h.matches("<tr class=\"pass\">").count(), 1);
assert!(h.contains("&lt;script&gt;") && !h.contains("<script>"), "{h}");
assert!(h.contains("not conformant"));
}
}

69
wai/wai-rs/src/bin/wai_cli/dx/conform/suites.rs
View file

@ -12,6 +12,11 @@
//! | packager | `policy` | `pack-conformance/` | a policy file | it is admitted with the declared digest, or refused with the declared reason (packaging §3, §12) |
//! | origin | `http` | `http-conformance/objects/` | an object | it is served as http-delivery §5.3 and §6.2 require: bytes, headers, ranges, head proof, preflight |
//! | sink | `native-sink` | `live-conformance/native-sink/vectors.json` | one session, its answers withheld, or one configuration | the answers the sink writes, one a line in RFC 8785 form, have the SHA-256 of the answers the corpus states (native-sink §10) |
//!
//! Each suite names its corpus's independent verifier where there is one (`claim`, `jwp`,
//! `signer` and `conform-result` case by case, through their `verdicts` mode; `http` and
//! `native-sink` over the whole corpus); the score and policy corpora have none here (the packaging corpus's,
//! `pack_verify.py`, runs in wai-pack.yml beside the reference packager).
use serde_json::Value;
use std::path::{Path, PathBuf};
@ -65,19 +70,23 @@ pub struct Suite {
pub http: bool,
pub command: bool,
pub load: fn(&Path) -> Result<Loaded, String>,
/// The corpus's independent verifier (`dx::independent`), which `--independent` runs beside
/// the implementation: case by case where it has a `verdicts` mode, over the whole corpus
/// otherwise; none where the corpus has no second implementation.
pub independent: Option<&'static str>,
}
pub const KINDS: &[&str] = &["verifier", "decoder", "packager", "origin", "sink"];
pub const SUITES: &[Suite] = &[
Suite { kind: "verifier", name: "claim", dir: "claim-conformance", about: "typed claim receipts (jwp-receipts §3.1): accept with the link hash, or reject at a step with a reason", http: true, command: true, load: claim },
Suite { kind: "verifier", name: "jwp", dir: "jwp-conformance", about: "labelled leaves, group receipts, totals, chains and the binary form (jwp-receipts §2–§6)", http: true, command: true, load: jwp },
Suite { kind: "verifier", name: "signer", dir: "signer-conformance", about: "key discovery, signing authority and module authorisation (jwp-receipts §7)", http: true, command: true, load: stated_lines },
Suite { kind: "verifier", name: "conform-result", dir: "conform-result-conformance", about: "conformance results, the signed record of a run (conformance-results §4)", http: true, command: true, load: stated_lines },
Suite { kind: "decoder", name: "score", dir: "score-conformance", about: "a WSCR score rendered to its canonical i16le PCM (the score extension's mixdown-equivalence)", http: true, command: true, load: score },
Suite { kind: "packager", name: "policy", dir: "pack-conformance", about: "packaging policies admitted with their digest, or refused with their reason (packaging §3, §12)", http: true, command: true, load: policy },
Suite { kind: "origin", name: "http", dir: "http-conformance", about: "objects, head proofs, ranges and headers as an origin serves them (http-delivery §5.3, §6.2)", http: true, command: false, load: http_origin },
Suite { kind: "sink", name: "native-sink", dir: "live-conformance/native-sink", about: "a native sink's answer to every call of a session, byte for byte, and its refusal of a configuration (native-sink §10)", http: true, command: true, load: native_sink },
Suite { kind: "verifier", name: "claim", dir: "claim-conformance", about: "typed claim receipts (jwp-receipts §3.1): accept with the link hash, or reject at a step with a reason", http: true, command: true, load: claim, independent: Some("claim") },
Suite { kind: "verifier", name: "jwp", dir: "jwp-conformance", about: "labelled leaves, group receipts, totals, chains and the binary form (jwp-receipts §2–§6)", http: true, command: true, load: jwp, independent: Some("jwp") },
Suite { kind: "verifier", name: "signer", dir: "signer-conformance", about: "key discovery, signing authority and module authorisation (jwp-receipts §7)", http: true, command: true, load: stated_lines, independent: Some("signer") },
Suite { kind: "verifier", name: "conform-result", dir: "conform-result-conformance", about: "conformance results, the signed record of a run (conformance-results §4)", http: true, command: true, load: stated_lines, independent: Some("conform-result") },
Suite { kind: "decoder", name: "score", dir: "score-conformance", about: "a WSCR score rendered to its canonical i16le PCM (the score extension's mixdown-equivalence)", http: true, command: true, load: score, independent: None },
Suite { kind: "packager", name: "policy", dir: "pack-conformance", about: "packaging policies admitted with their digest, or refused with their reason (packaging §3, §12)", http: true, command: true, load: policy, independent: None },
Suite { kind: "origin", name: "http", dir: "http-conformance", about: "objects, head proofs, ranges and headers as an origin serves them (http-delivery §5.3, §6.2)", http: true, command: false, load: http_origin, independent: Some("http") },
Suite { kind: "sink", name: "native-sink", dir: "live-conformance/native-sink", about: "a native sink's answer to every call of a session, byte for byte, and its refusal of a configuration (native-sink §10)", http: true, command: true, load: native_sink, independent: Some("native-sink") },
];
fn read_json(p: &Path) -> Result<Value, String> {
@ -297,6 +306,50 @@ mod tests {
use super::*;
use serde_json::json;
/// The rule `independent` follows: a suite names its corpus's own verifier where the table
/// holds one ([`crate::dx::independent::own_verifiers`]), and none where it holds none. The
/// findings, for a table of suites.
fn unnamed_verifiers(suites: &[Suite]) -> Vec<String> {
let mut out = Vec::new();
for s in suites {
let own: Vec<&str> = crate::dx::independent::own_verifiers(s.dir).iter().map(|v| v.name).collect();
match s.independent {
None if !own.is_empty() => out.push(format!("{}: its corpus {} has an independent verifier ({}), and the suite names none", s.name, s.dir, own.join(", "))),
Some(n) if !own.contains(&n) => out.push(format!("{}: {n} is not {}'s own verifier (the table's are: {})", s.name, s.dir, own.join(", "))),
_ => {}
}
}
out
}
#[test]
fn every_suite_names_its_corpuss_verifier_where_there_is_one() {
let found = unnamed_verifiers(SUITES);
assert!(found.is_empty(), "{found:#?}");
// The suites with one and without, as the module's text says.
let with: Vec<&str> = SUITES.iter().filter(|s| s.independent.is_some()).map(|s| s.name).collect();
assert_eq!(with, vec!["claim", "jwp", "signer", "conform-result", "http", "native-sink"]);
// The rule finds a suite that names none where its corpus has one.
let mut without: Vec<Suite> = SUITES.iter().map(|s| Suite { independent: s.independent, ..*s }).collect();
without.iter_mut().find(|s| s.name == "http").unwrap().independent = None;
assert_eq!(unnamed_verifiers(&without).len(), 1, "{:?}", unnamed_verifiers(&without));
}
#[test]
fn every_suites_independent_verifier_is_an_entry_with_the_mode_it_needs() {
for s in SUITES {
if let Some(n) = s.independent {
let v = crate::dx::independent::named(n).unwrap_or_else(|| panic!("{}: {n} is not an independent verifier", s.name));
// A verifier suite is held case by case: its corpus file is the verdicts mode's.
if s.kind == "verifier" {
let root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).parent().unwrap();
let loaded = (s.load)(&root.join(s.dir)).unwrap();
assert_eq!(v.verdicts.map(|f| root.join(f)), loaded.corpus_file, "{}: the independent verifier reads the corpus file the suite gives", s.name);
}
}
}
}
#[test]
fn verdict_lines_follow_the_corpora() {
assert_eq!(claim_expected(&json!({"outcome": "reject", "step": 3, "reason": "signature_invalid"})).0, "reject step=3 reason=signature_invalid");

Some files were not shown because too many files have changed in this diff Show more