Transaction-Science/open-standards
Watch
1
0
Fork
You've already forked open-standards
 0
Code Issues Pull requests Projects Releases 17 Packages Wiki Activity Actions 33
open-standards/wai/conform-result-conformance/vectors.json
Transaction Science f33765a846 conform-result-conformance: each summary count wrong alone, its other counts right 
The falsification of the reference found that leaving out the check of summary.cases
alone left the corpus passing: every summary vector had passed and failed wrong too.
Three vectors now carry each count wrong alone.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-27 11:17:06 -04:00

147 lines
34 KiB
JSON
Raw Permalink Blame History

{
"about": "conformance-results §7: records of conform-result/1, each with the outcome §4 gives it. Signed with the published test keys of seeds 32 bytes of 0x43 and of 0x44, which attest nothing; every run, program and path in them is an example.",
"public_keys": [
"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6",
"d759793bbc13a2819a827c76adb6fba8a49aee007f49f2d0992d99b825ad2c48"
],
"vectors": [
{
"name": "valid",
"about": "a conformant run, every case passed, from the command adapter with a key made for the run",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":true,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"reject step=3 reason=bad_signer_key\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":3,\"failed\":0},\"conformant\":true,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"e9dd68e9c855e98fe4c7a1221429fb48712d5cd14d96a9bd05c74e92577df3d8f531c9c274cb6c024103dd8569d5f0ed8fc4199929e78dc05dcfbdc17d07a804\"}",
"expected": "accept conformant=true"
},
{
"name": "valid_not_conformant",
"about": "a run with one failed case: a genuine record of a run that did not conform",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":false,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"accept\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":2,\"failed\":1},\"conformant\":false,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"988b7c0a1141b8793f049c84a64033f6138be7c0898a5ac9f13ad0f83055a8533dc1778586a4b31edadef53a20947cd0787b0fc194b01ab5cb156484cb400a0c\"}",
"expected": "accept conformant=false"
},
{
"name": "valid_no_cases",
"about": "a run with no case: not conformant",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[],\"summary\":{\"cases\":0,\"passed\":0,\"failed\":0},\"conformant\":false,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"efcb2c1b3554fa1115990d121acb0545011df768cdc517c48c9d4e0b100da19316b3082b32d9aa497478e5a25eb86c0e156943f156152cb5ac4053752066990f\"}",
"expected": "accept conformant=false"
},
{
"name": "valid_http_supplied_key",
"about": "a conformant run of a decoder from the HTTP adapter, signed with a key the runner holds, its name and version given",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"decoder\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"http\",\"impl\":\"http://127.0.0.1:8380/verify\",\"name\":\"an example verifier\",\"version\":\"1.0\",\"files\":[]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":true,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"reject step=3 reason=bad_signer_key\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":3,\"failed\":0},\"conformant\":true,\"key\":\"supplied\",\"signer_pubkey\":\"d759793bbc13a2819a827c76adb6fba8a49aee007f49f2d0992d99b825ad2c48\",\"sig\":\"74a2f68e0bd8a05a9ad772a3ffd4f3a90e39d8aacc177d545e9d35a95c44119d8ac0db1f29c65f9d06c2c3fc512e1f013ec50173d39970a710932eed03c1110c\"}",
"expected": "accept conformant=true"
},
{
"name": "flipped_case",
"about": "the failed case turned into a pass after signing, its summary and verdict made to agree",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":true,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"accept\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":3,\"failed\":0},\"conformant\":true,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"988b7c0a1141b8793f049c84a64033f6138be7c0898a5ac9f13ad0f83055a8533dc1778586a4b31edadef53a20947cd0787b0fc194b01ab5cb156484cb400a0c\"}",
"expected": "reject step=4 reason=signature_invalid"
},
{
"name": "corpus_digest_changed",
"about": "the corpus digest's last digit changed after signing",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad0\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":true,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"reject step=3 reason=bad_signer_key\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":3,\"failed\":0},\"conformant\":true,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"e9dd68e9c855e98fe4c7a1221429fb48712d5cd14d96a9bd05c74e92577df3d8f531c9c274cb6c024103dd8569d5f0ed8fc4199929e78dc05dcfbdc17d07a804\"}",
"expected": "reject step=4 reason=signature_invalid"
},
{
"name": "summary_changed",
"about": "the summary alone changed after signing",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":false,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"accept\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":2,\"failed\":0},\"conformant\":false,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"988b7c0a1141b8793f049c84a64033f6138be7c0898a5ac9f13ad0f83055a8533dc1778586a4b31edadef53a20947cd0787b0fc194b01ab5cb156484cb400a0c\"}",
"expected": "reject step=4 reason=signature_invalid"
},
{
"name": "bad_signature",
"about": "one bit of the signature changed",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":true,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"reject step=3 reason=bad_signer_key\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":3,\"failed\":0},\"conformant\":true,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"e8dd68e9c855e98fe4c7a1221429fb48712d5cd14d96a9bd05c74e92577df3d8f531c9c274cb6c024103dd8569d5f0ed8fc4199929e78dc05dcfbdc17d07a804\"}",
"expected": "reject step=4 reason=signature_invalid"
},
{
"name": "signature_s_not_reduced",
"about": "the signature's S replaced by S + L, which verifies under the cofactorless equation reduced mod L but not under the strict rule",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":true,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"reject step=3 reason=bad_signer_key\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":3,\"failed\":0},\"conformant\":true,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"e9dd68e9c855e98fe4c7a1221429fb48712d5cd14d96a9bd05c74e92577df3d8e205bf1f8f2e7f5a17a0d42848cfcf0290c4199929e78dc05dcfbdc17d07a814\"}",
"expected": "reject step=4 reason=signature_invalid"
},
{
"name": "key_small_order",
"about": "the identity point as the key with (R, S) = (B, 1), which the cofactorless equation accepts over any message",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":true,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"reject step=3 reason=bad_signer_key\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":3,\"failed\":0},\"conformant\":true,\"key\":\"ephemeral\",\"signer_pubkey\":\"0100000000000000000000000000000000000000000000000000000000000000\",\"sig\":\"58666666666666666666666666666666666666666666666666666666666666660100000000000000000000000000000000000000000000000000000000000000\"}",
"expected": "reject step=4 reason=signature_invalid"
},
{
"name": "summary_mismatch_signed",
"about": "signed as it is: a summary that does not count its cases",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":false,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"accept\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":3,\"failed\":0},\"conformant\":false,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"a817be73b9f129adf5e67e48ea19ea5ea122df1bcce00caac8fd06a4154f319786cbe6e26382402bc3a3f30b6f584ffea6decff2f755f468aba68ec76ac2f200\"}",
"expected": "reject step=5 reason=summary_mismatch"
},
{
"name": "summary_cases_mismatch_signed",
"about": "signed as it is: a case count one more than its cases, its passed and failed right",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":false,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"accept\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":4,\"passed\":2,\"failed\":1},\"conformant\":false,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"5d30d9eb3b01a65112f048bbe9dcdf85abef30cbea3e5d8832c09237a51a0189fd262e1ee9aa5f10488be40259b75cddb06550b2b641cc5fee1d5872ab80d907\"}",
"expected": "reject step=5 reason=summary_mismatch"
},
{
"name": "summary_passed_mismatch_signed",
"about": "signed as it is: a passed count one more than its cases pass, its other counts right",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":false,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"accept\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":3,\"failed\":1},\"conformant\":false,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"e77942ca2e7278a07eb78fe94e9f808c211e726d4b01d8d2d8b10ff8c5711dcae25038296d6a11a1c3a14cd31e71e85409e16775d5c71e5b9ef9c3392b473207\"}",
"expected": "reject step=5 reason=summary_mismatch"
},
{
"name": "summary_failed_mismatch_signed",
"about": "signed as it is: a failed count one less than its cases fail, its other counts right",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":false,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"accept\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":2,\"failed\":0},\"conformant\":false,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"2f3270617bb5b746f4357175afcd7b0096aeb28a21cb8553cfcc5cb20dea07f38e4deda292447e8cf33ec5276d6f8c361c1aece305afbbbd07a999f9fe71630c\"}",
"expected": "reject step=5 reason=summary_mismatch"
},
{
"name": "conformant_mismatch_signed",
"about": "signed as it is: conformant though a case failed",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":false,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"accept\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":2,\"failed\":1},\"conformant\":true,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"9a822a7581ce6828e53a8010b3e74af8042b9fa3e75aa093760f5f483fba3fd27ea7fde36c4f5f2615ab10a486ef62bd22150642733aefda27182f1170745700\"}",
"expected": "reject step=6 reason=conformant_mismatch"
},
{
"name": "conformant_no_cases_signed",
"about": "signed as it is: conformant with no case",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[],\"summary\":{\"cases\":0,\"passed\":0,\"failed\":0},\"conformant\":true,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"ff8802bb7268c58e149e6e3a83ed90cda92b5d6b89d59658e80a274ff250624998136d3924139e30f6880a5e5f6df9996522204a3df73b0623368298da5c7500\"}",
"expected": "reject step=6 reason=conformant_mismatch"
},
{
"name": "unknown_format",
"about": "signed as it is: a version this revision does not define",
"record": "{\"wai\":\"conform-result/2\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":true,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"reject step=3 reason=bad_signer_key\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":3,\"failed\":0},\"conformant\":true,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"9b004322af6b5d17316e311765435a037c531ad4617df6504fe3ce5c2cd6a8c01bfee4c56676115ae2e39bb18e456583ec600bd78c1ba009cba78af74663710c\"}",
"expected": "reject step=2 reason=unknown_format"
},
{
"name": "missing_member",
"about": "signed as it is: no corpus member",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"key\":\"ephemeral\",\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":true,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"reject step=3 reason=bad_signer_key\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":3,\"failed\":0},\"conformant\":true,\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"aaaeec52c150ea472a45ef47b4a54f152fd2c2d6a79e9abb2397f409b6ff4eccb4c360c4c9413685140b7426a4b9df6f444d7a2fd67a45d445ba2de6cbe9dd06\"}",
"expected": "reject step=3 reason=malformed"
},
{
"name": "extra_member",
"about": "signed as it is: a member §2 does not list",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":true,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"reject step=3 reason=bad_signer_key\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":3,\"failed\":0},\"conformant\":true,\"key\":\"ephemeral\",\"note\":\"a member §2 does not list\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"6ca3491e6a018d64a9fdb5fa0eaa096a6e6b744d5c6646e4b4d2d690a6e3c943b137023b39fbd7ba3bf0bd51157e99f729411228851443d69700f0c8303ed600\"}",
"expected": "reject step=3 reason=malformed"
},
{
"name": "duplicate_member",
"about": "the valid record with its verdict named twice, the second time true",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":true,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"reject step=3 reason=bad_signer_key\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":3,\"failed\":0},\"conformant\":false,\"conformant\":true,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"e9dd68e9c855e98fe4c7a1221429fb48712d5cd14d96a9bd05c74e92577df3d8f531c9c274cb6c024103dd8569d5f0ed8fc4199929e78dc05dcfbdc17d07a804\"}",
"expected": "reject step=1 reason=malformed"
},
{
"name": "fraction_count",
"about": "the valid record with a count written with a fraction (3.0)",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":true,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"reject step=3 reason=bad_signer_key\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3.0,\"passed\":3,\"failed\":0},\"conformant\":true,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"e9dd68e9c855e98fe4c7a1221429fb48712d5cd14d96a9bd05c74e92577df3d8f531c9c274cb6c024103dd8569d5f0ed8fc4199929e78dc05dcfbdc17d07a804\"}",
"expected": "reject step=3 reason=malformed"
},
{
"name": "digest_upper_case",
"about": "the valid record with its corpus digest in upper-case hexadecimal",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360CE9739BEFC69B6773ED9FC087B2A9611EB75910E62796EA1B886770DAD9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":true,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"reject step=3 reason=bad_signer_key\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":3,\"failed\":0},\"conformant\":true,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"e9dd68e9c855e98fe4c7a1221429fb48712d5cd14d96a9bd05c74e92577df3d8f531c9c274cb6c024103dd8569d5f0ed8fc4199929e78dc05dcfbdc17d07a804\"}",
"expected": "reject step=3 reason=malformed"
},
{
"name": "not_json",
"about": "bytes that are not a JSON text",
"record": "{\"wai\":\"conform-result/1\",\"kind\":\"verifier\",\"suite\":\"claim\",\"standard\":\"v0.4.0\",\"toolchain\":{\"name\":\"wai\",\"version\":\"0.3.0\"},\"corpus\":{\"dir\":\"claim-conformance\",\"files\":1,\"digest\":\"sha256:50360ce9739befc69b6773ed9fc087b2a9611eb75910e62796ea1b886770dad9\"},\"implementation\":{\"adapter\":\"command\",\"impl\":\"example-verifier verdicts {corpus}\",\"name\":null,\"version\":null,\"files\":[{\"path\":\"/opt/example/bin/example-verifier\",\"sha256\":\"6b0cf49408a2a43a27d45717c016625528f9e09892b99827c8a3aaefe1aed964\"}]},\"started_at\":\"2026-10-27T12:00:00Z\",\"cases\":[{\"name\":\"signature_invalid\",\"pass\":true,\"expected\":\"reject step=3 reason=signature_invalid\",\"got\":\"reject step=3 reason=signature_invalid\"},{\"name\":\"bad_signer_key\",\"pass\":true,\"expected\":\"reject step=3 reason=bad_signer_key\",\"got\":\"reject step=3 reason=bad_signer_key\"},{\"name\":\"field_type\",\"pass\":true,\"expected\":\"reject step=3 reason=field_type\",\"got\":\"reject step=3 reason=field_type\"}],\"summary\":{\"cases\":3,\"passed\":3,\"failed\":0},\"conformant\":true,\"key\":\"ephemeral\",\"signer_pubkey\":\"22fc297792f0b6ffc0bfcfdb7edb0c0aa14e025a365ec0e342e86e3829cb74b6\",\"sig\":\"e9dd68e9c855e98fe4c7a1221429fb48712d5cd14d96a9bd05c74e92577df3d8f531c9c274cb6c024103dd8569d5f0ed8fc4199929e78dc05dcfbdc17d07a804\"",
"expected": "reject step=1 reason=malformed"
}
]
}
Reference in a new issue View git blame Copy permalink