From 02c98eac84d4eee85b77e0c248e351ecf0d276e3 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Sat, 19 Sep 2026 05:35:38 -0400 Subject: [PATCH 01/12] smart-byte: verify what the pack really contains, and pin what it does not MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `smart-byte/conformance/v1` is published for third parties to certify against. Its gate — `.github/workflows/conformance.yml`, named "smart-byte conformance gate" — runs `jq` over it: valid JSON, top-level array, `name` and `description` present, names unique. Those are well-formedness checks. No value in the pack has ever been compared against the implementation, and `smart-byte-rs` has no conformance crate that reads it. `conformance/v1/regeneration.md` is candid that the `expected_*` fields are placeholders "not yet cryptographically computed". So this is a known state, openly documented, and not a defect anyone hid. What has changed is its stated precondition: the doc says regenerate "when `smart-byte-rs` lands", describing it as a "private repo, to be open-sourced alongside spec v1.0 freeze". `smart-byte-rs` HAS landed — 22 crates, in the `standards-build` matrix — so the one mechanical pass that doc describes is now possible, and overdue. Its named entry point, `cargo run --bin gen-conformance-vectors`, does not exist. What the pack actually contains, established by running it rather than reading it: REAL — every positive vector's `signing_key_seed` derives exactly the `verifying_key` it publishes. That is asserted properly here, along with `sig-004-wrong-key`'s deliberate mismatch, so the negative case cannot quietly become a positive one. NOT REAL — the `expected_*` values: - `sig-001` says it signs "a known envelope SAID". It does not. The value is RFC 8032's signature over the EMPTY message, proven by verifying it against `b""`. `sig-002`/`sig-003` carry another RFC 8032 signature, over neither their SAID nor the empty message. - Every SAID envelope carries the literal ``; every `expected_said` is one repeated character rather than a digest; every `canonical_cbor_hex` is the same 7-byte truncated prefix. - `canon-002` and `canon-003` publish `expected_cbor_hex` of ODD length (109 and 37 characters), which cannot be decoded to bytes at all. A third party fails before reaching any comparison. - The doc's own consistency claim does not hold in the direction that matters: `said-004` and `said-005` are different envelopes published with the SAME expected SAID, so even as filler the pack asserts a collision. The pinning tests are written to FAIL once the pack is regenerated, which is the intended signal to replace each with a real assertion — the same pattern `map-conformance` uses for its JSON round-trip defect. Each was falsified by simulating the fix: substituting a real-looking SAID, giving `said-005` a distinct value, making `canon-002`'s hex even-length, corrupting a verifying key, and replacing `sig-001`'s signature. All five fail as intended, and the pack was restored and verified byte-identical to main. Regenerating a published pack is a normative change to a public artifact and a maintainer's call, so no vector is touched here. This only makes the state visible in test output instead of in a doc nobody opens. Co-Authored-By: Claude Opus 5 --- smart-byte/smart-byte-rs/Cargo.lock | 2 + .../crates/smart-byte-core/Cargo.toml | 4 + .../crates/smart-byte-core/tests/pack.rs | 209 ++++++++++++++++++ 3 files changed, 215 insertions(+) create mode 100644 smart-byte/smart-byte-rs/crates/smart-byte-core/tests/pack.rs diff --git a/smart-byte/smart-byte-rs/Cargo.lock b/smart-byte/smart-byte-rs/Cargo.lock index 2f1681b85..ebcc32cb5 100644 --- a/smart-byte/smart-byte-rs/Cargo.lock +++ b/smart-byte/smart-byte-rs/Cargo.lock @@ -5950,10 +5950,12 @@ dependencies = [ "chrono", "data-encoding", "ed25519-dalek 2.2.0", + "hex", "rand 0.8.6", "serde", "serde_bytes", "serde_cbor", + "serde_json", "thiserror 2.0.18", "uuid", ] diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-core/Cargo.toml b/smart-byte/smart-byte-rs/crates/smart-byte-core/Cargo.toml index 5624c2a9d..675379f81 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-core/Cargo.toml +++ b/smart-byte/smart-byte-rs/crates/smart-byte-core/Cargo.toml @@ -18,3 +18,7 @@ thiserror = { workspace = true } uuid = { workspace = true } data-encoding = { workspace = true } rand = { workspace = true } + +[dev-dependencies] +serde_json = "1" +hex = "0.4" diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-core/tests/pack.rs b/smart-byte/smart-byte-rs/crates/smart-byte-core/tests/pack.rs new file mode 100644 index 000000000..2d3a4485e --- /dev/null +++ b/smart-byte/smart-byte-rs/crates/smart-byte-core/tests/pack.rs @@ -0,0 +1,209 @@ +//! Hold `smart-byte/conformance/v1` to what it actually contains. +//! +//! The pack is published for third parties to certify against, and its gate — +//! `.github/workflows/conformance.yml`, named "smart-byte conformance gate" — runs +//! `jq` over it: valid JSON, top-level array, every entry has `name` and +//! `description`, names unique within a file. Those are well-formedness checks. **No +//! value in the pack has ever been compared against the implementation**, and there +//! is no conformance crate in `smart-byte-rs` that reads it. +//! +//! `conformance/v1/regeneration.md` is candid that the `expected_*` fields are +//! placeholders "not yet cryptographically computed", to be regenerated "when +//! `smart-byte-rs` lands". So this is a known state, not a discovered defect. What +//! has changed is the precondition: `smart-byte-rs` **has** landed — 22 crates, in +//! the `standards-build` matrix — so the mechanical pass that doc describes is now +//! possible and overdue. (Its named entry point, `cargo run --bin +//! gen-conformance-vectors`, does not exist yet.) +//! +//! This file therefore does two different jobs, and says which is which: +//! +//! 1. **Verifies what is real.** The `signing_key_seed` → `verifying_key` +//! derivations are genuine Ed25519 and are checked against the implementation's +//! own re-export. +//! 2. **Pins what is not**, so the placeholder state is visible in test output +//! rather than only in a doc nobody opens. Those tests are written to **FAIL +//! once the pack is regenerated**, which is the intended signal to replace them +//! with real assertions — the same pattern `map-conformance` uses for its JSON +//! round-trip defect. +//! +//! Regenerating the pack is a normative change to a published artifact and is a +//! maintainer's call, so nothing here changes a vector. + +use smart_byte_core::sign::{Signature, SigningKey, Verifier, VerifyingKey}; + +fn pack(name: &str) -> serde_json::Value { + let p = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("../../../conformance/v1") + .canonicalize() + .expect("the conformance pack must exist next to the spec") + .join(name); + serde_json::from_slice(&std::fs::read(&p).unwrap_or_else(|e| panic!("read {name}: {e}"))) + .unwrap_or_else(|e| panic!("{name} must parse: {e}")) +} + +fn vectors(name: &str) -> Vec { + pack(name).as_array().expect("a vector file is a JSON array").clone() +} + +/// REAL: every positive vector's seed derives exactly the verifying key it publishes. +/// +/// This is the one cryptographic relationship in the pack that holds today, so it is +/// asserted properly rather than pinned. `sig-004-wrong-key` is the deliberate +/// exception — its key is *meant* not to match, and that is asserted too, so the +/// negative case cannot quietly become a positive one. +#[test] +fn published_signing_seeds_derive_their_published_verifying_keys() { + let mut checked = 0; + for v in vectors("signature_vectors.json") { + let name = v["name"].as_str().expect("vector has a name"); + let (Some(seed), Some(vk_hex)) = + (v["signing_key_seed"].as_str(), v["verifying_key"].as_str()) + else { + continue; + }; + let bytes: [u8; 32] = hex::decode(seed) + .expect("seed is hex") + .try_into() + .expect("an Ed25519 seed is 32 bytes"); + let derived = hex::encode(SigningKey::from_bytes(&bytes).verifying_key().to_bytes()); + + if name == "sig-004-wrong-key" { + assert_ne!( + derived, vk_hex, + "sig-004 is the wrong-key negative case; its key must NOT derive from its seed" + ); + } else { + assert_eq!(derived, vk_hex, "{name}: published verifying key is not this seed's"); + checked += 1; + } + } + assert_eq!(checked, 4, "expected 4 positive signature vectors"); +} + +/// PINNED DEFECT: the published signatures are not signatures over the SAIDs they name. +/// +/// `sig-001`'s description says it signs "a known envelope SAID". It does not: the +/// value is RFC 8032's test signature over the **empty message**, which is proven +/// here by verifying it against `b""`. `sig-002`/`sig-003` carry another RFC 8032 +/// signature, over neither their SAID nor the empty message. +/// +/// **This test is written to fail once the pack is regenerated.** When it does, +/// replace it with the real assertion: each signature verifies over its own +/// `envelope_said`. +#[test] +fn the_published_signatures_do_not_yet_sign_their_stated_saids() { + let vs = vectors("signature_vectors.json"); + + let one = vs.iter().find(|v| v["name"] == "sig-001-known-key-known-said").expect("sig-001"); + let vk = VerifyingKey::from_bytes( + &hex::decode(one["verifying_key"].as_str().unwrap()).unwrap().try_into().unwrap(), + ) + .expect("a valid Ed25519 key"); + let sig = Signature::from_bytes( + &hex::decode(one["signature"].as_str().unwrap()).unwrap().try_into().unwrap(), + ); + let said = one["envelope_said"].as_str().unwrap(); + + assert!( + vk.verify(said.as_bytes(), &sig).is_err(), + "sig-001 now signs its SAID — the pack was regenerated; replace this test with \ + the real assertion that every signature verifies over its envelope_said" + ); + assert!( + vk.verify(b"", &sig).is_ok(), + "sig-001 is no longer RFC 8032's empty-message signature, but still does not sign \ + its SAID — the pack changed into a third state that neither assertion describes" + ); +} + +/// PINNED DEFECT: the SAID vectors are filler, and say so on their face. +/// +/// Every envelope carries the literal string ``, every +/// `expected_said` is a repeated-character pattern rather than a digest, and every +/// `canonical_cbor_hex` is the same 7-byte prefix rather than a complete encoding. +/// +/// **Written to fail once regenerated.** +#[test] +fn the_published_said_vectors_are_still_placeholders() { + let vs = vectors("said_vectors.json"); + assert_eq!(vs.len(), 5, "the SAID vector count changed"); + + for v in &vs { + let name = v["name"].as_str().unwrap(); + assert_eq!( + v["envelope"]["said"].as_str(), + Some(""), + "{name}: the envelope no longer carries the placeholder marker — regenerated?" + ); + + // A real SAID is BLAKE3-256 in base32; filler is one character repeated. + let said = v["expected_said"].as_str().expect("expected_said is a string"); + let body = &said[1..]; + assert!( + body.chars().all(|c| c == body.as_bytes()[0] as char), + "{name}: expected_said is no longer a repeated-character pattern — replace this \ + test with a real `Envelope::compute_said()` comparison" + ); + + assert_eq!( + v["canonical_cbor_hex"].as_str().map(str::len), + Some(14), + "{name}: canonical_cbor_hex is no longer the 7-byte truncated prefix" + ); + } +} + +/// PINNED DEFECT: the pack's own internal-consistency claim does not hold. +/// +/// `regeneration.md` states the placeholders are "internally consistent (the same +/// envelope produces the same SAID across vectors)". The converse fails: two +/// *different* envelopes are given the *same* expected SAID, so even as filler the +/// pack asserts a collision. Any implementation deriving SAIDs from envelopes would +/// have to produce one too. +#[test] +fn two_different_envelopes_are_published_with_the_same_said() { + let vs = vectors("said_vectors.json"); + let four = vs.iter().find(|v| v["name"] == "said-004-with-provenance").expect("said-004"); + let five = + vs.iter().find(|v| v["name"] == "said-005-with-ownership-transitions").expect("said-005"); + + assert_ne!(four["envelope"], five["envelope"], "said-004 and said-005 are the same envelope"); + assert_eq!( + four["expected_said"], five["expected_said"], + "the duplicate SAID is fixed — replace this test with a real uniqueness assertion" + ); +} + +/// PINNED DEFECT: two canonicalisation vectors are not decodable hex. +/// +/// `canon-002` and `canon-003` publish `expected_cbor_hex` strings of **odd** length +/// (109 and 37 characters), which cannot be decoded to bytes at all. A third party +/// following the spec fails before reaching any comparison. +/// +/// **Written to fail once fixed.** +#[test] +fn two_canonicalisation_vectors_publish_undecodable_hex() { + let mut odd: Vec<(String, usize)> = Vec::new(); + for v in vectors("canonicalisation_vectors.json") { + let h = v["expected_cbor_hex"].as_str().expect("expected_cbor_hex is a string"); + if h.len() % 2 == 1 { + odd.push((v["name"].as_str().unwrap().to_string(), h.len())); + } + // Whatever its length, it must at least be hex characters. + assert!( + h.bytes().all(|b| b.is_ascii_hexdigit()), + "{}: expected_cbor_hex contains non-hex characters", + v["name"] + ); + } + odd.sort(); + assert_eq!( + odd, + vec![ + ("canon-002-shortest-form-integers".to_string(), 109), + ("canon-003-no-tags".to_string(), 37), + ], + "the set of undecodable canonicalisation vectors changed — if they were fixed, \ + replace this test with a real canonical-CBOR comparison" + ); +} -- 2.54.0 (Apple Git-157) From fb8552a72cfec4461d13aa84c0578cff65ecd549 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Mon, 21 Sep 2026 15:20:15 -0400 Subject: [PATCH 02/12] smart-byte: the spec mandates a vector suite that does not exist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to the pack audit in this branch, from asking why the vectors could not simply be regenerated. The answer is that the spec is in worse shape than the pack. `spec/identity_and_key_rotation.md` says implementers **"MUST pass every vector"** in the SAID pack. Every `expected_said` in that pack is a repeated-character placeholder rather than a digest, so the requirement is unsatisfiable as written — a conforming implementation cannot be conforming. The same sentence describes the suite as covering "the empty-map case, the one-field case, the worked example above, several envelope-shaped objects from treatise §8, and one adversarial case where a non-canonical CBOR encoding produces a different digest (to verify that implementations reject non-canonical input rather than silently re-encoding)". The published file holds five vectors, all envelope-shaped. **The adversarial case is the only negative test named, it is the security-relevant one, and it is absent** — an implementation that silently re-encodes non-canonical input passes this pack completely. Pinned by `the_spec_mandates_a_suite_the_published_pack_does_not_contain`, written to fail once the pack is regenerated, which is the signal to replace it with a real assertion that the adversarial vector exists and is rejected. Falsified both ways: adding an adversarial vector fails it, and softening the spec's MUST fails it. Pack and spec restored and verified clean afterwards. Also fixes a broken reference, which is a correction rather than a normative change. The spec pointed three times at `../conformance/said_vectors.json`, which does not exist — the file is at `../conformance/v1/said_vectors.json`. Line 266 states "all paths are relative to this file", so the breakage was unambiguous and so is the intent. An implementer following the normative reference reached nothing. Nothing else in the spec or the pack is touched. Regenerating the vectors, and deciding whether the pack should gain the missing cases, stays a maintainer's call. Workspace 622 passed, 0 failures; clippy clean; `check-no-vanity-metrics.sh` passes. Co-Authored-By: Claude Opus 5 --- .../crates/smart-byte-core/tests/pack.rs | 53 +++++++++++++++++++ smart-byte/spec/identity_and_key_rotation.md | 6 +-- 2 files changed, 56 insertions(+), 3 deletions(-) diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-core/tests/pack.rs b/smart-byte/smart-byte-rs/crates/smart-byte-core/tests/pack.rs index 2d3a4485e..711db9030 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-core/tests/pack.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-core/tests/pack.rs @@ -207,3 +207,56 @@ fn two_canonicalisation_vectors_publish_undecodable_hex() { replace this test with a real canonical-CBOR comparison" ); } + +/// PINNED DEFECT: the spec makes a **MUST** out of vectors nothing can pass, and +/// describes a suite the published file does not contain. +/// +/// `spec/identity_and_key_rotation.md` says implementers "MUST pass every vector" +/// in the SAID pack. Every `expected_said` in that pack is a repeated-character +/// placeholder rather than a digest, so the requirement is unsatisfiable as written +/// — a conforming implementation cannot be conforming. +/// +/// The same sentence describes a suite of "the empty-map case, the one-field case, +/// the worked example above, several envelope-shaped objects from treatise §8, and +/// one adversarial case where a non-canonical CBOR encoding produces a different +/// digest (to verify that implementations reject non-canonical input rather than +/// silently re-encoding)". The published file holds five vectors, all +/// envelope-shaped. **The adversarial case is the security-relevant one — it is the +/// only negative test named — and it is absent.** An implementation that silently +/// re-encodes non-canonical input would pass this pack completely. +/// +/// **Written to fail once the pack is regenerated**, which is the signal to replace +/// this with a real assertion that the adversarial vector is present and rejected. +#[test] +fn the_spec_mandates_a_suite_the_published_pack_does_not_contain() { + let spec = std::fs::read_to_string( + std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("../../../spec/identity_and_key_rotation.md"), + ) + .expect("the identity spec must sit beside the pack"); + + assert!( + spec.contains("MUST pass every vector"), + "the spec no longer mandates the pack; this pin needs rewriting" + ); + assert!( + spec.contains("non-canonical CBOR encoding produces a different digest"), + "the spec no longer describes an adversarial case; this pin needs rewriting" + ); + + let vs = vectors("said_vectors.json"); + let adversarial: Vec<&str> = vs + .iter() + .filter(|v| { + let hay = format!("{} {}", v["name"], v["description"]).to_lowercase(); + hay.contains("non-canonical") || hay.contains("adversarial") || hay.contains("reject") + }) + .map(|v| v["name"].as_str().unwrap_or("")) + .collect(); + + assert!( + adversarial.is_empty(), + "the adversarial non-canonical vector the spec mandates now exists ({adversarial:?}) — \ + replace this test with one that feeds it non-canonical CBOR and requires rejection" + ); +} diff --git a/smart-byte/spec/identity_and_key_rotation.md b/smart-byte/spec/identity_and_key_rotation.md index 08b2963f5..1667d8f0a 100644 --- a/smart-byte/spec/identity_and_key_rotation.md +++ b/smart-byte/spec/identity_and_key_rotation.md @@ -106,7 +106,7 @@ Any verifier in possession of the object replaces `d` with 53 `#` characters, re ### 2.5 Test-vector hook -The full set of canonical worked examples — placeholder bytes, canonical CBOR bytes, BLAKE3 digest, final SAID — lives in `../conformance/said_vectors.json` (created in issue #6). Implementers MUST pass every vector. The suite covers the empty-map case, the one-field case, the worked example above, several envelope-shaped objects from treatise §8, and one adversarial case where a non-canonical CBOR encoding produces a different digest (to verify that implementations reject non-canonical input rather than silently re-encoding). +The full set of canonical worked examples — placeholder bytes, canonical CBOR bytes, BLAKE3 digest, final SAID — lives in `../conformance/v1/said_vectors.json` (created in issue #6). Implementers MUST pass every vector. The suite covers the empty-map case, the one-field case, the worked example above, several envelope-shaped objects from treatise §8, and one adversarial case where a non-canonical CBOR encoding produces a different digest (to verify that implementations reject non-canonical input rather than silently re-encoding). --- @@ -248,7 +248,7 @@ The Rust reference implementation of this section lives in the `smart-byte-rs` w - `smart-byte-rs/crates/smart-byte-core/src/said.rs` — SAID computation, placeholder substitution, canonical CBOR encoding, per-key SAID derivation, and the conformance-vector harness referenced in §2.5. - `smart-byte-rs/crates/smart-byte-core/src/keri.rs` — key-event log: event type definitions, the wrapper that pairs events with signatures, the verifier algorithm in §4, and helpers for emitting `icp`, `rot`, and `ixn` events from an in-memory controller state. -If these modules are not yet present in `smart-byte-rs` at the time this section lands, the implementer should treat the function signatures and field-name conventions in §§2–4 as the v1 interface contract and implement against them directly; the reference crate will conform to those signatures when it lands. The conformance test vectors in `../conformance/said_vectors.json` (issue #6) are the authoritative reference, independent of any single language's implementation. +If these modules are not yet present in `smart-byte-rs` at the time this section lands, the implementer should treat the function signatures and field-name conventions in §§2–4 as the v1 interface contract and implement against them directly; the reference crate will conform to those signatures when it lands. The conformance test vectors in `../conformance/v1/said_vectors.json` (issue #6) are the authoritative reference, independent of any single language's implementation. --- @@ -263,4 +263,4 @@ If these modules are not yet present in `smart-byte-rs` at the time this section - O'Connor, Jack, et al. *BLAKE3: One Function, Fast Everywhere*. 2020. - Global Legal Entity Identifier Foundation (GLEIF). *vLEI Ecosystem Governance Framework*. Production deployments via Provenant and Veridian, 2024–2026. -Cross-references within this spec are to other files in `./` (e.g. [`treatise_v1_parts_I_II_III_combined.md`](./treatise_v1_parts_I_II_III_combined.md), `../conformance/said_vectors.json`); all paths are relative to this file. +Cross-references within this spec are to other files in `./` (e.g. [`treatise_v1_parts_I_II_III_combined.md`](./treatise_v1_parts_I_II_III_combined.md), `../conformance/v1/said_vectors.json`); all paths are relative to this file. -- 2.54.0 (Apple Git-157) From 5391e895d58dcaa9d22dc193c152c14ecffa26cc Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Tue, 22 Sep 2026 01:19:24 -0400 Subject: [PATCH 03/12] smart-byte: record the editorial reason for the spec reference fix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI caught this correctly and I had missed it. `conformance.yml`'s coupling gate requires a change under `smart-byte/spec/` to be accompanied by a change under `smart-byte/conformance/`, and the previous commit fixed three broken references in the spec without touching the pack. The gate's own message gives the route for an editorial change: record the reason in `conformance/v/regeneration.md` and include that file. This does that. The wire format, the vectors and every `expected_*` value remain untouched. The pack was not edited to satisfy the gate — only this log was added, which is what the gate asks for. Two further findings went into the same entry, since regeneration.md is where whoever regenerates will look, and neither is fixed here because both are normative: - **The regeneration procedure is not runnable as written.** It names `cargo run --bin gen-conformance-vectors` to produce the pack and `verify-conformance-vectors` to check it. NEITHER binary exists — the workspace ships one binary, `smart-byte`. So the mechanical pass this document describes cannot be run in either direction, which is stronger than "it has not been run yet". I had previously found only the first of the two. - **The precondition for regenerating has been met.** The status section says the placeholders stand until `smart-byte-rs` lands and calls it a private repo to be open-sourced at spec v1.0 freeze. It has landed and is in the `standards-build` matrix, so the blocker recorded there is stale. The entry also carries forward the missing adversarial non-canonical CBOR vector, so that a regeneration which replaces placeholder values without adding it does not quietly leave the spec describing a suite the pack lacks. `check-no-vanity-metrics.sh` passes; pack tests green. The coupling gate was simulated locally against its own logic before pushing (spec_touched=1, conformance_touched=1). Co-Authored-By: Claude Opus 5 --- smart-byte/conformance/v1/regeneration.md | 43 +++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/smart-byte/conformance/v1/regeneration.md b/smart-byte/conformance/v1/regeneration.md index 9cb8c381e..b70a4169f 100644 --- a/smart-byte/conformance/v1/regeneration.md +++ b/smart-byte/conformance/v1/regeneration.md @@ -106,3 +106,46 @@ Go and TypeScript) must independently produce the same `expected_*` values from the same inputs. If implementations disagree, the disagreement is resolved by appeal to the spec text, not by which implementation emitted the value first. + +## Editorial change log + +### 2026-09-21 — spec reference corrected; no wire-format change + +`spec/identity_and_key_rotation.md` pointed three times at +`../conformance/said_vectors.json`. That path does not resolve: the pack is at +`../conformance/v1/said_vectors.json`. Line 266 of that spec states "all paths +are relative to this file", so both the breakage and the intent were +unambiguous, and an implementer following the normative reference reached +nothing. The three references now carry the `v1/` segment. + +**The wire format, the vectors and every `expected_*` value are unchanged.** +This entry exists because the conformance gate requires a spec change to be +accompanied by a pack change, and directs an editorial change to record its +reason here. Recording it is the point; nothing in the pack was edited to +satisfy the gate. + +Two things found while making that correction, neither of them fixed here +because both are normative decisions: + +1. **The regeneration procedure above is not runnable as written.** It names + `cargo run --bin gen-conformance-vectors` to produce the pack and + `verify-conformance-vectors` to check it. **Neither binary exists** in + `smart-byte-rs` — the workspace ships one binary, `smart-byte`. So the pass + this document describes cannot be run in either direction, which is a + stronger statement than "it has not been run yet". + +2. **The precondition this document gives for regenerating has been met.** The + status section says the placeholders stand until `smart-byte-rs` lands, and + describes it as a private repo to be open-sourced at spec v1.0 freeze. It has + landed, it is in this repository, and it is built and tested by the + `standards-build` CI matrix. The blocker recorded here is stale. + +Also worth carrying into whoever does the regeneration: +`spec/identity_and_key_rotation.md` says implementers **MUST pass every vector**, +and describes the suite as including "one adversarial case where a non-canonical +CBOR encoding produces a different digest (to verify that implementations reject +non-canonical input rather than silently re-encoding)". **That vector is not in +this pack** — every SAID vector here is envelope-shaped, and the adversarial one +is the only negative case the spec names. A regeneration that replaces the placeholder +values without adding it would still leave the spec describing a suite this pack +does not contain. -- 2.54.0 (Apple Git-157) From 464a40777b18d80c4fa3569ccac65e5c97addac7 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Wed, 30 Sep 2026 20:30:04 -0400 Subject: [PATCH 04/12] smart-byte-core: canonical CBOR and the identity spec's SAIDs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Identity spec §2 defines a SAID precisely: hash the canonical CBOR (§2.3) of the object with its SAID field holding 53 `#`, and write the BLAKE3-256 digest as `E` + lowercase unpadded base32. Nothing in smart-byte-rs did that. Envelope ids and the key-event log hash `serde_cbor` output — struct-declaration key order, `[u8; 32]` written as arrays of small integers — over a zeroed id, and print upper-case base32 with no derivation code. The conformance pack could not be generated from any of it. - `canonical`: a CBOR data model with no float or tag variant, an encoder that sorts map keys by their encoded bytes (not their text), and a STRICT decoder that refuses every non-canonical form with its own error class instead of re-encoding it. Re-encoding is exactly what §2.5's adversarial vector exists to catch: it would let two byte strings verify under one SAID. - `said`: the §2.2 text form (one spelling only: lowercase, `E`, zero trailing bits), `derive`, `verify` and the §3.4 per-key SAID. The module docs now say plainly which identifiers in this crate follow §2 and which do not yet, and the two contradictory doc comments about the text form's case are gone. - `sign`: `sign_said` / `verify_said_signature`, verifying strictly (S in [0, L), no small-order points) and reporting malformed input before any cryptography runs. Existing APIs are untouched; `Said` additionally derives `Ord`. Co-Authored-By: Claude Opus 5.5 --- .../crates/smart-byte-core/src/canonical.rs | 540 ++++++++++++++++++ .../crates/smart-byte-core/src/lib.rs | 8 +- .../crates/smart-byte-core/src/said.rs | 274 ++++++++- .../crates/smart-byte-core/src/sign.rs | 126 ++++ 4 files changed, 940 insertions(+), 8 deletions(-) create mode 100644 smart-byte/smart-byte-rs/crates/smart-byte-core/src/canonical.rs diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/canonical.rs b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/canonical.rs new file mode 100644 index 000000000..735860f14 --- /dev/null +++ b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/canonical.rs @@ -0,0 +1,540 @@ +//! Canonical CBOR — the encoding every SAID is computed over. +//! +//! Identity spec §2.3 fixes one byte form per value: RFC 8949 §4.2.1 core +//! deterministic encoding, with floats, tags and indefinite lengths forbidden. +//! [`encode`] produces that form from a [`Value`]. [`decode`] accepts **only** +//! that form: a strict decoder that refuses non-canonical bytes instead of +//! re-encoding them, which is the property the spec's adversarial SAID vector +//! (§2.5) exists to test. Re-encoding would make a non-canonical copy of an +//! object hash to the canonical SAID, so two different byte strings would both +//! verify under one identifier. +//! +//! The rules, and the error class [`decode`] reports for each: +//! +//! | rule (§2.3 / RFC 8949) | class | +//! | -------------------------------------------------------- | ---------------------- | +//! | integers use the shortest head | `non_shortest_integer` | +//! | string, array and map lengths use the shortest head | `non_shortest_length` | +//! | map keys sorted by their encoded bytes, not their text | `unsorted_map_keys` | +//! | no key twice in one map | `duplicate_map_key` | +//! | no floats | `float_forbidden` | +//! | no tags | `tag_forbidden` | +//! | no indefinite-length items | `indefinite_length` | +//! | text strings are UTF-8 | `invalid_utf8` | +//! | the input is exactly one item | `trailing_bytes` | +//! +//! Inputs that are not well-formed CBOR at all report `truncated` or +//! `malformed`. Nesting deeper than [`MAX_DEPTH`] reports `too_deep`. That +//! limit is this implementation's guard against unbounded recursion, not a +//! canonical-form rule. + +use std::cmp::Ordering; + +/// How deeply [`decode`] lets arrays and maps nest before refusing the input. +pub const MAX_DEPTH: usize = 128; + +/// A value in the CBOR data model that canonical encoding admits. +/// +/// There is no float and no tag variant, because neither has a canonical form. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Value { + /// Major type 0: `0 ..= u64::MAX`. + Unsigned(u64), + /// Major type 1: the integer `-1 - n`, so `Negative(0)` is `-1`. + Negative(u64), + /// Major type 2. + Bytes(Vec), + /// Major type 3. + Text(String), + /// Major type 4. + Array(Vec), + /// Major type 5. Entries may be held in any order; [`encode`] sorts them. + Map(Vec<(Value, Value)>), + /// Simple values 20 and 21. + Bool(bool), + /// Simple value 22. + Null, + /// Any other simple value: `0 ..= 19`, `23` (undefined), `32 ..= 255`. + /// `24 ..= 31` have no well-formed encoding (RFC 8949 §3.3). + Simple(u8), +} + +impl Value { + /// A signed integer, placed in major type 0 or 1 as its sign requires. + pub fn int(i: i64) -> Self { + if i >= 0 { + Value::Unsigned(i as u64) + } else { + // -1 - n = i => n = -1 - i, which cannot overflow for i < 0. + Value::Negative((-1 - i) as u64) + } + } + + /// A text value. + pub fn text(s: impl Into) -> Self { + Value::Text(s.into()) + } + + /// A map from text keys, in the order given. + pub fn map>(entries: impl IntoIterator) -> Self { + Value::Map( + entries + .into_iter() + .map(|(k, v)| (Value::Text(k.into()), v)) + .collect(), + ) + } + + /// The value held under text key `key`, if this is a map holding one. + pub fn get(&self, key: &str) -> Option<&Value> { + match self { + Value::Map(entries) => entries + .iter() + .find(|(k, _)| matches!(k, Value::Text(t) if t == key)) + .map(|(_, v)| v), + _ => None, + } + } + + /// Mutable access to the value held under text key `key`. + pub fn get_mut(&mut self, key: &str) -> Option<&mut Value> { + match self { + Value::Map(entries) => entries + .iter_mut() + .find(|(k, _)| matches!(k, Value::Text(t) if t == key)) + .map(|(_, v)| v), + _ => None, + } + } +} + +/// Why an encoding was refused. [`CanonicalError::class`] gives the stable +/// name the conformance pack uses for each. +#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] +pub enum CanonicalError { + #[error("input ends inside an item")] + Truncated, + #[error("bytes follow the top-level item")] + TrailingBytes, + #[error("not well-formed CBOR")] + Malformed, + #[error("integer not in its shortest form")] + NonShortestInteger, + #[error("length not in its shortest form")] + NonShortestLength, + #[error("indefinite-length item")] + IndefiniteLength, + #[error("CBOR tags are forbidden")] + TagForbidden, + #[error("floating-point values are forbidden")] + FloatForbidden, + #[error("map keys are not in canonical order")] + UnsortedMapKeys, + #[error("a map key appears twice")] + DuplicateMapKey, + #[error("text string is not valid UTF-8")] + InvalidUtf8, + #[error("nested deeper than the decoder allows")] + TooDeep, +} + +impl CanonicalError { + /// The error class as the conformance pack names it. + pub fn class(&self) -> &'static str { + match self { + CanonicalError::Truncated => "truncated", + CanonicalError::TrailingBytes => "trailing_bytes", + CanonicalError::Malformed => "malformed", + CanonicalError::NonShortestInteger => "non_shortest_integer", + CanonicalError::NonShortestLength => "non_shortest_length", + CanonicalError::IndefiniteLength => "indefinite_length", + CanonicalError::TagForbidden => "tag_forbidden", + CanonicalError::FloatForbidden => "float_forbidden", + CanonicalError::UnsortedMapKeys => "unsorted_map_keys", + CanonicalError::DuplicateMapKey => "duplicate_map_key", + CanonicalError::InvalidUtf8 => "invalid_utf8", + CanonicalError::TooDeep => "too_deep", + } + } +} + +/// Encode `value` canonically. +/// +/// Fails only for values that have no canonical form: a map holding the same +/// key twice (`duplicate_map_key`) or a simple value in `24 ..= 31` +/// (`malformed`). +pub fn encode(value: &Value) -> Result, CanonicalError> { + let mut out = Vec::new(); + encode_into(value, &mut out)?; + Ok(out) +} + +fn head(out: &mut Vec, major: u8, arg: u64) { + let m = major << 5; + if arg < 24 { + out.push(m | arg as u8); + } else if arg <= 0xff { + out.push(m | 24); + out.push(arg as u8); + } else if arg <= 0xffff { + out.push(m | 25); + out.extend_from_slice(&(arg as u16).to_be_bytes()); + } else if arg <= 0xffff_ffff { + out.push(m | 26); + out.extend_from_slice(&(arg as u32).to_be_bytes()); + } else { + out.push(m | 27); + out.extend_from_slice(&arg.to_be_bytes()); + } +} + +fn encode_into(value: &Value, out: &mut Vec) -> Result<(), CanonicalError> { + match value { + Value::Unsigned(n) => head(out, 0, *n), + Value::Negative(n) => head(out, 1, *n), + Value::Bytes(b) => { + head(out, 2, b.len() as u64); + out.extend_from_slice(b); + } + Value::Text(t) => { + head(out, 3, t.len() as u64); + out.extend_from_slice(t.as_bytes()); + } + Value::Array(items) => { + head(out, 4, items.len() as u64); + for item in items { + encode_into(item, out)?; + } + } + Value::Map(entries) => { + let mut encoded = Vec::with_capacity(entries.len()); + for (k, v) in entries { + encoded.push((encode(k)?, encode(v)?)); + } + // §2.3: "sorted lexicographically by their canonical CBOR byte + // representation, *not* by the corresponding string". + encoded.sort_by(|a, b| a.0.cmp(&b.0)); + if encoded.windows(2).any(|w| w[0].0 == w[1].0) { + return Err(CanonicalError::DuplicateMapKey); + } + head(out, 5, encoded.len() as u64); + for (k, v) in encoded { + out.extend_from_slice(&k); + out.extend_from_slice(&v); + } + } + Value::Bool(false) => out.push(0xf4), + Value::Bool(true) => out.push(0xf5), + Value::Null => out.push(0xf6), + Value::Simple(n) => match *n { + 0..=23 => out.push(0xe0 | *n), + 24..=31 => return Err(CanonicalError::Malformed), + _ => { + out.push(0xf8); + out.push(*n); + } + }, + } + Ok(()) +} + +/// Decode `bytes`, accepting only the canonical encoding of exactly one value. +pub fn decode(bytes: &[u8]) -> Result { + let mut d = Decoder { buf: bytes, pos: 0 }; + let value = d.item(0)?; + if d.pos != bytes.len() { + return Err(CanonicalError::TrailingBytes); + } + Ok(value) +} + +struct Decoder<'a> { + buf: &'a [u8], + pos: usize, +} + +impl<'a> Decoder<'a> { + fn take(&mut self, n: usize) -> Result<&'a [u8], CanonicalError> { + let end = self.pos.checked_add(n).ok_or(CanonicalError::Truncated)?; + let s = self + .buf + .get(self.pos..end) + .ok_or(CanonicalError::Truncated)?; + self.pos = end; + Ok(s) + } + + fn byte(&mut self) -> Result { + Ok(self.take(1)?[0]) + } + + /// The argument of a head whose additional information is `ai`, refusing + /// any head longer than the argument needs. + fn argument(&mut self, ai: u8, non_shortest: CanonicalError) -> Result { + let (arg, floor) = match ai { + 0..=23 => return Ok(u64::from(ai)), + 24 => (u64::from(self.byte()?), 24), + 25 => ( + u64::from(u16::from_be_bytes(self.take(2)?.try_into().unwrap())), + 0x100, + ), + 26 => ( + u64::from(u32::from_be_bytes(self.take(4)?.try_into().unwrap())), + 0x1_0000, + ), + 27 => ( + u64::from_be_bytes(self.take(8)?.try_into().unwrap()), + 0x1_0000_0000, + ), + _ => unreachable!("additional information 28..=31 is handled by the caller"), + }; + if arg < floor { + Err(non_shortest) + } else { + Ok(arg) + } + } + + fn length(&mut self, ai: u8) -> Result { + let n = self.argument(ai, CanonicalError::NonShortestLength)?; + // A length larger than the input cannot be satisfied; refusing it here + // also keeps an adversarial length from driving a huge allocation. + let n = usize::try_from(n).map_err(|_| CanonicalError::Truncated)?; + if n > self.buf.len() - self.pos { + return Err(CanonicalError::Truncated); + } + Ok(n) + } + + fn item(&mut self, depth: usize) -> Result { + if depth > MAX_DEPTH { + return Err(CanonicalError::TooDeep); + } + let initial = self.byte()?; + let major = initial >> 5; + let ai = initial & 0x1f; + + if major == 7 { + return match ai { + 0..=19 | 23 => Ok(Value::Simple(ai)), + 20 => Ok(Value::Bool(false)), + 21 => Ok(Value::Bool(true)), + 22 => Ok(Value::Null), + 24 => match self.byte()? { + // RFC 8949 §3.3: a two-byte simple value below 32 is not + // well-formed (it would have a one-byte encoding). + n if n < 32 => Err(CanonicalError::Malformed), + n => Ok(Value::Simple(n)), + }, + 25..=27 => Err(CanonicalError::FloatForbidden), + // 28..=30 are reserved; 31 is a "break" with nothing to end. + _ => Err(CanonicalError::Malformed), + }; + } + if major == 6 { + return Err(CanonicalError::TagForbidden); + } + match ai { + 28..=30 => return Err(CanonicalError::Malformed), + 31 => { + return Err(if major <= 1 { + CanonicalError::Malformed + } else { + CanonicalError::IndefiniteLength + }); + } + _ => {} + } + + match major { + 0 => Ok(Value::Unsigned( + self.argument(ai, CanonicalError::NonShortestInteger)?, + )), + 1 => Ok(Value::Negative( + self.argument(ai, CanonicalError::NonShortestInteger)?, + )), + 2 => { + let n = self.length(ai)?; + Ok(Value::Bytes(self.take(n)?.to_vec())) + } + 3 => { + let n = self.length(ai)?; + let raw = self.take(n)?; + let s = std::str::from_utf8(raw).map_err(|_| CanonicalError::InvalidUtf8)?; + Ok(Value::Text(s.to_owned())) + } + 4 => { + let n = self.length(ai)?; + let mut items = Vec::with_capacity(n); + for _ in 0..n { + items.push(self.item(depth + 1)?); + } + Ok(Value::Array(items)) + } + 5 => { + // Every entry is at least two bytes, so `length` bounding the + // count by the remaining input bounds it twice over. + let n = self.length(ai)?; + let mut entries = Vec::with_capacity(n); + let mut previous_key: Option<&'a [u8]> = None; + for _ in 0..n { + let start = self.pos; + let key = self.item(depth + 1)?; + // The decoder is strict, so the bytes it just consumed ARE + // the key's canonical encoding: order is checked on those. + let key_bytes = &self.buf[start..self.pos]; + if let Some(prev) = previous_key { + match prev.cmp(key_bytes) { + Ordering::Less => {} + Ordering::Equal => return Err(CanonicalError::DuplicateMapKey), + Ordering::Greater => return Err(CanonicalError::UnsortedMapKeys), + } + } + previous_key = Some(key_bytes); + let value = self.item(depth + 1)?; + entries.push((key, value)); + } + Ok(Value::Map(entries)) + } + _ => unreachable!("major types 6 and 7 are handled above"), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn hex(s: &str) -> Vec { + (0..s.len()) + .step_by(2) + .map(|i| u8::from_str_radix(&s[i..i + 2], 16).unwrap()) + .collect() + } + + fn enc(v: &Value) -> String { + encode(v) + .unwrap() + .iter() + .map(|b| format!("{b:02x}")) + .collect() + } + + #[test] + fn integers_take_the_shortest_head_at_every_boundary() { + for (n, want) in [ + (0u64, "00"), + (23, "17"), + (24, "1818"), + (255, "18ff"), + (256, "190100"), + (65535, "19ffff"), + (65536, "1a00010000"), + (4294967295, "1affffffff"), + (4294967296, "1b0000000100000000"), + (u64::MAX, "1bffffffffffffffff"), + ] { + assert_eq!(enc(&Value::Unsigned(n)), want, "{n}"); + assert_eq!(decode(&hex(want)).unwrap(), Value::Unsigned(n)); + } + assert_eq!(enc(&Value::int(-1)), "20"); + assert_eq!(enc(&Value::int(-24)), "37"); + assert_eq!(enc(&Value::int(-25)), "3818"); + assert_eq!(enc(&Value::int(i64::MIN)), "3b7fffffffffffffff"); + } + + #[test] + fn map_keys_sort_by_encoded_bytes_not_by_text() { + // As strings "aa" < "b"; encoded, 0x61 0x62 ("b") < 0x62 0x61 0x61. + let v = Value::map([("aa", Value::int(1)), ("b", Value::int(2))]); + assert_eq!(enc(&v), "a261620262616101"); + } + + #[test] + fn the_spec_worked_example_is_61_bytes() { + let v = Value::map([("n", Value::int(1)), ("d", Value::text("#".repeat(53)))]); + let bytes = encode(&v).unwrap(); + assert_eq!(bytes.len(), 61); + assert_eq!(&bytes[..5], &[0xa2, 0x61, 0x64, 0x78, 0x35]); + assert_eq!(&bytes[58..], &[0x61, 0x6e, 0x01]); + } + + #[test] + fn duplicate_keys_have_no_canonical_form() { + let v = Value::map([("a", Value::int(1)), ("a", Value::int(2))]); + assert_eq!(encode(&v), Err(CanonicalError::DuplicateMapKey)); + } + + #[test] + fn decode_round_trips_everything_encode_produces() { + let v = Value::map([ + ("bytes", Value::Bytes(vec![0, 1, 2])), + ("text", Value::text("ünïcode")), + ( + "list", + Value::Array(vec![Value::Null, Value::Bool(true), Value::Simple(23)]), + ), + ("nested", Value::map([("x", Value::int(-1000))])), + ("hi", Value::Simple(255)), + ]); + let bytes = encode(&v).unwrap(); + let back = decode(&bytes).unwrap(); + assert_eq!(encode(&back).unwrap(), bytes); + } + + #[test] + fn every_non_canonical_form_is_refused_with_its_own_class() { + use CanonicalError::*; + for (h, want) in [ + ("1800", NonShortestInteger), + ("190017", NonShortestInteger), + ("3800", NonShortestInteger), + ("1b00000000ffffffff", NonShortestInteger), + ("780161", NonShortestLength), + ("5801ff", NonShortestLength), + ("980100", NonShortestLength), + ("a262616101616202", UnsortedMapKeys), + ("a2616101616102", DuplicateMapKey), + ("9f0102ff", IndefiniteLength), + ("7f6161ff", IndefiniteLength), + ("bf616101ff", IndefiniteLength), + ("d54100", TagForbidden), + ("f93e00", FloatForbidden), + ("fa7f7fffff", FloatForbidden), + ("fb7fefffffffffffff", FloatForbidden), + ("62c328", InvalidUtf8), + ("0101", TrailingBytes), + ("1a0001", Truncated), + ("43", Truncated), + ("", Truncated), + ("1c", Malformed), + ("ff", Malformed), + ("f81f", Malformed), + ("1f", Malformed), + ] { + assert_eq!(decode(&hex(h)), Err(want), "{h}"); + } + } + + #[test] + fn nesting_is_bounded() { + let mut deep = vec![0x81u8; MAX_DEPTH + 1]; + deep.push(0x00); + assert_eq!(decode(&deep), Err(CanonicalError::TooDeep)); + let mut ok = vec![0x81u8; MAX_DEPTH]; + ok.push(0x00); + assert!(decode(&ok).is_ok()); + } + + #[test] + fn a_huge_declared_length_is_refused_without_allocating() { + assert_eq!( + decode(&hex("5bffffffffffffffff")), + Err(CanonicalError::Truncated) + ); + assert_eq!( + decode(&hex("9bffffffffffffffff")), + Err(CanonicalError::Truncated) + ); + } +} diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/lib.rs b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/lib.rs index c3224cb15..a0a4a92a6 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/lib.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/lib.rs @@ -6,6 +6,9 @@ //! * [`Envelope`] — content-addressed signed carrier of arbitrary cargo //! with intrinsic provenance, ownership history, and energy cost. //! * [`Said`] — Self-Addressing IDentifier (BLAKE3 over canonical CBOR). +//! [`said::derive`] / [`said::verify`] implement identity spec §2 over +//! [`canonical`] CBOR; see the [`said`] module for which identifiers in this +//! crate do and do not follow it yet. //! * [`sign`] / [`verify`] — Ed25519 signature over the envelope's SAID. //! * [`Cargo`] — discriminated union of byte payload, USD claim, //! joule claim, or custom typed body. @@ -20,6 +23,7 @@ //! language can therefore compute the same SAID for the same //! semantic envelope without trusting each other. +pub mod canonical; pub mod cargo; pub mod envelope; pub mod joule_cost; @@ -33,8 +37,8 @@ pub use envelope::{Envelope, EnvelopeError}; pub use joule_cost::JouleCost; pub use ownership::{OwnershipChain, Transition}; pub use provenance::Provenance; -pub use said::{Said, SaidError}; -pub use sign::{Signature, SigningKey, VerifyingKey, sign, verify}; +pub use said::{Said, SaidCheckError, SaidDerivation, SaidError}; +pub use sign::{SaidSignatureError, Signature, SigningKey, VerifyingKey, sign, verify}; /// BLAKE3 32-byte hash output. Re-exported so downstream crates can /// refer to it without depending on `blake3` directly. diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/said.rs b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/said.rs index 8df04af8d..dff7244e7 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/said.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/said.rs @@ -1,14 +1,40 @@ //! Self-Addressing IDentifiers. //! -//! A [`Said`] is a 32-byte BLAKE3 digest, displayed as bytewise base32 -//! (RFC 4648, lower-case, no padding). The substrate uses SAIDs for -//! envelope ids and for principal identities. +//! A [`Said`] is a 32-byte BLAKE3 digest. It has two text forms, and they are +//! not interchangeable: +//! +//! * **The identity spec's form** (§2.2): the derivation code `E` followed by +//! the digest in *lowercase* unpadded base32 — 53 characters. +//! [`Said::to_said_text`] / [`Said::from_said_text`] produce and parse it, +//! and [`derive`] / [`verify`] implement the §2.1 procedure that yields it: +//! hash the canonical CBOR ([`crate::canonical`]) of the object with its +//! SAID field holding [`SAID_PLACEHOLDER`]. The conformance pack in +//! `smart-byte/conformance/v1` is generated from these functions. +//! * **The older form** that [`Display`](fmt::Display) and +//! [`Said::to_base32`] print: *upper-case* unpadded base32 with no +//! derivation code. It predates the identity spec. [`Envelope`] ids and the +//! key-event log in `smart-byte-keri-witness` still derive and print SAIDs +//! this way (`serde_cbor` over a zeroed id), so their identifiers are not +//! §2 SAIDs yet. +//! +//! [`Envelope`]: crate::Envelope use std::fmt; use data_encoding::BASE32_NOPAD; use serde::{Deserialize, Serialize}; +use crate::canonical::{self, CanonicalError, Value}; + +/// The derivation code of a BLAKE3-256 SAID (identity spec §2.2). +pub const BLAKE3_256_CODE: char = 'E'; + +/// Characters in a SAID's text form: the code plus 52 base32 characters. +pub const SAID_TEXT_LEN: usize = 53; + +/// What a SAID field holds while its object is hashed: `#` 53 times (§2.2). +pub const SAID_PLACEHOLDER: &str = "#####################################################"; + /// Errors returned when parsing a [`Said`] from text. #[derive(Debug, thiserror::Error)] pub enum SaidError { @@ -20,9 +46,8 @@ pub enum SaidError { /// 32-byte self-addressing identifier. /// -/// Equality is byte-equality. The textual form is upper-case base32 -/// without padding so it can be embedded in URLs and copied by humans. -#[derive(Clone, Copy, PartialEq, Eq, Hash, Default, Serialize, Deserialize)] +/// Equality is byte-equality. See the module docs for its two text forms. +#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Default, Serialize, Deserialize)] #[serde(transparent)] pub struct Said(pub [u8; 32]); @@ -56,6 +81,149 @@ impl Said { out.copy_from_slice(&bytes); Ok(Self(out)) } + + /// The identity spec's text form: `E` + lowercase unpadded base32 (§2.2). + pub fn to_said_text(&self) -> String { + let mut s = String::with_capacity(SAID_TEXT_LEN); + s.push(BLAKE3_256_CODE); + s.push_str(&BASE32_NOPAD.encode(&self.0).to_ascii_lowercase()); + s + } + + /// Parse the identity spec's text form. + /// + /// Only the one canonical spelling is accepted: exactly 53 characters, + /// the `E` code, lowercase base32, and zero trailing bits in the last + /// character. Anything looser would give one digest several spellings, + /// and a SAID compared as text would then fail to match itself. + pub fn from_said_text(s: &str) -> Result { + let body = s + .strip_prefix(BLAKE3_256_CODE) + .filter(|b| { + b.len() == SAID_TEXT_LEN - 1 + && b.bytes().all(|c| matches!(c, b'a'..=b'z' | b'2'..=b'7')) + }) + .ok_or_else(|| { + SaidError::Decode(format!( + "not a {SAID_TEXT_LEN}-character `E` + lowercase base32 SAID: {s:?}" + )) + })?; + // `from_base32` decodes with trailing-bit checking, which is what + // refuses a non-zero final character. + Self::from_base32(body) + } +} + +/// The steps of one SAID derivation (identity spec §2.1), each kept so a test +/// vector can publish all of them. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SaidDerivation { + /// Canonical CBOR of the object with its SAID field holding + /// [`SAID_PLACEHOLDER`] — the bytes that are hashed. + pub hashed: Vec, + /// BLAKE3-256 of `hashed`. + pub said: Said, + /// The object with the SAID in its field. + pub object: Value, + /// Canonical CBOR of `object`. + pub encoded: Vec, +} + +/// Why a §2 derivation or verification refused its input. +#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] +pub enum SaidCheckError { + #[error("not canonical CBOR: {0}")] + NonCanonical(#[from] CanonicalError), + #[error("a SAID-bearing object must be a map")] + NotAMap, + #[error("the object has no `{0}` field")] + MissingSaidField(String), + #[error("the `{0}` field does not hold a well-formed SAID")] + MalformedSaid(String), + #[error("the object states SAID {stated} but hashes to {computed}")] + Mismatch { stated: String, computed: String }, +} + +impl SaidCheckError { + /// The error class as the conformance pack names it. + pub fn class(&self) -> &'static str { + match self { + SaidCheckError::NonCanonical(e) => e.class(), + SaidCheckError::NotAMap => "not_a_map", + SaidCheckError::MissingSaidField(_) => "missing_said_field", + SaidCheckError::MalformedSaid(_) => "malformed_said", + SaidCheckError::Mismatch { .. } => "said_mismatch", + } + } +} + +/// Derive the SAID of `object`, which carries its SAID in the member named +/// `field` (identity spec §2.1). Whatever that member holds now is replaced by +/// the placeholder before hashing, so deriving over an already-stamped object +/// gives the same SAID as deriving over a fresh one. +pub fn derive(object: &Value, field: &str) -> Result { + if !matches!(object, Value::Map(_)) { + return Err(SaidCheckError::NotAMap); + } + let mut stamped = object.clone(); + let slot = stamped + .get_mut(field) + .ok_or_else(|| SaidCheckError::MissingSaidField(field.to_owned()))?; + *slot = Value::text(SAID_PLACEHOLDER); + let hashed = canonical::encode(&stamped)?; + let said = Said::hash(&hashed); + if let Some(slot) = stamped.get_mut(field) { + *slot = Value::Text(said.to_said_text()); + } + let encoded = canonical::encode(&stamped)?; + Ok(SaidDerivation { + hashed, + said, + object: stamped, + encoded, + }) +} + +/// Verify received bytes against the SAID they state in `field`. +/// +/// The bytes must already be canonical: a non-canonical copy is refused with +/// the [`CanonicalError`] it violates, never re-encoded and accepted. That is +/// the behaviour identity spec §2.5's adversarial vector tests — re-encoding +/// would let two different byte strings verify under one SAID. +pub fn verify(bytes: &[u8], field: &str) -> Result { + let object = canonical::decode(bytes)?; + if !matches!(object, Value::Map(_)) { + return Err(SaidCheckError::NotAMap); + } + let stated = match object.get(field) { + None => return Err(SaidCheckError::MissingSaidField(field.to_owned())), + Some(Value::Text(t)) => { + Said::from_said_text(t).map_err(|_| SaidCheckError::MalformedSaid(field.to_owned()))? + } + Some(_) => return Err(SaidCheckError::MalformedSaid(field.to_owned())), + }; + let computed = derive(&object, field)?.said; + if computed != stated { + return Err(SaidCheckError::Mismatch { + stated: stated.to_said_text(), + computed: computed.to_said_text(), + }); + } + Ok(stated) +} + +/// The per-key SAID of identity spec §3.4: the SAID of +/// `{"d": , "k": }`, where `key` is the key's algorithm +/// identifier byte (§3.3; `0x01` = Ed25519) followed by the public key, carried +/// as a byte string. +pub fn per_key_said(prefixed_key: &[u8]) -> Said { + let object = Value::map([ + ("d", Value::text(SAID_PLACEHOLDER)), + ("k", Value::Bytes(prefixed_key.to_vec())), + ]); + derive(&object, "d") + .expect("a map with two distinct text keys always has a canonical form") + .said } impl fmt::Display for Said { @@ -94,4 +262,98 @@ mod tests { let err = Said::from_base32("AAAA").unwrap_err(); assert!(matches!(err, SaidError::WrongLength(_))); } + + #[test] + fn spec_text_form_is_53_lowercase_characters_and_round_trips() { + let said = Said::hash(b"smart-byte"); + let text = said.to_said_text(); + assert_eq!(text.len(), SAID_TEXT_LEN); + assert!(text.starts_with('E')); + assert!( + text[1..] + .bytes() + .all(|c| matches!(c, b'a'..=b'z' | b'2'..=b'7')) + ); + assert_eq!(Said::from_said_text(&text).unwrap(), said); + } + + #[test] + fn spec_text_form_has_exactly_one_spelling() { + let text = Said::hash(b"x").to_said_text(); + // Upper case, a missing or wrong code, and a wrong length are refused. + assert!(Said::from_said_text(&text.to_ascii_uppercase()).is_err()); + assert!(Said::from_said_text(&text[1..]).is_err()); + assert!(Said::from_said_text(&format!("F{}", &text[1..])).is_err()); + assert!(Said::from_said_text(&text[..52]).is_err()); + // The 52nd base32 character carries 4 padding bits that must be zero: + // of its 32 possible values only 'a' (0) and 'q' (16) are canonical. + let mut body = "a".repeat(51); + body.push('b'); + assert!(Said::from_said_text(&format!("E{body}")).is_err()); + body.pop(); + body.push('q'); + assert!(Said::from_said_text(&format!("E{body}")).is_ok()); + } + + fn worked_example() -> Value { + Value::map([("d", Value::text(SAID_PLACEHOLDER)), ("n", Value::int(1))]) + } + + #[test] + fn derive_hashes_the_placeholder_form_and_stamps_the_result() { + let d = derive(&worked_example(), "d").unwrap(); + assert_eq!(d.hashed.len(), 61); + assert_eq!(d.said, Said::hash(&d.hashed)); + assert_eq!( + d.encoded.len(), + d.hashed.len(), + "the placeholder keeps the length stable" + ); + assert_eq!(verify(&d.encoded, "d").unwrap(), d.said); + // Deriving again over the stamped object gives the same SAID. + assert_eq!(derive(&d.object, "d").unwrap().said, d.said); + } + + #[test] + fn verify_refuses_changed_content_and_non_canonical_copies() { + let d = derive(&worked_example(), "d").unwrap(); + + // Change n from 1 to 2 and keep the stated SAID. + let mut changed = d.encoded.clone(); + *changed.last_mut().unwrap() = 0x02; + assert_eq!(verify(&changed, "d").unwrap_err().class(), "said_mismatch"); + + // The same object with n spelled 0x18 0x01: refused as non-canonical, + // even though re-encoding it would reproduce the stated SAID. + let mut lax = d.encoded.clone(); + lax.pop(); + lax.extend_from_slice(&[0x18, 0x01]); + assert_eq!( + verify(&lax, "d").unwrap_err().class(), + "non_shortest_integer" + ); + } + + #[test] + fn verify_names_each_structural_failure() { + let not_a_map = canonical::encode(&Value::int(1)).unwrap(); + assert_eq!(verify(¬_a_map, "d").unwrap_err().class(), "not_a_map"); + let no_field = canonical::encode(&Value::map([("n", Value::int(1))])).unwrap(); + assert_eq!( + verify(&no_field, "d").unwrap_err().class(), + "missing_said_field" + ); + let bad = canonical::encode(&Value::map([("d", Value::text("Enot-a-said"))])).unwrap(); + assert_eq!(verify(&bad, "d").unwrap_err().class(), "malformed_said"); + } + + #[test] + fn per_key_said_commits_to_the_algorithm_byte() { + let key = [7u8; 32]; + let mut ed = vec![0x01]; + ed.extend_from_slice(&key); + let mut other = vec![0x02]; + other.extend_from_slice(&key); + assert_ne!(per_key_said(&ed), per_key_said(&other)); + } } diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/sign.rs b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/sign.rs index 7c93bd4ac..33a78b1f6 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/sign.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/sign.rs @@ -9,6 +9,7 @@ use ed25519_dalek::Signer; pub use ed25519_dalek::{Signature, SigningKey, Verifier, VerifyingKey}; use crate::envelope::Envelope; +use crate::said::Said; /// Errors returned by verification. #[derive(Debug, thiserror::Error)] @@ -32,3 +33,128 @@ pub fn verify( key.verify(envelope.id.as_bytes(), sig) .map_err(|e| SignError::Bad(e.to_string())) } + +/// Sign a SAID. The message is the 32 digest bytes the SAID encodes, not its +/// 53-character text — the same bytes [`sign`] signs for an envelope. +pub fn sign_said(said: &Said, key: &SigningKey) -> Signature { + key.sign(said.as_bytes()) +} + +/// Why a signature over a SAID was refused. +#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] +pub enum SaidSignatureError { + #[error("not a well-formed SAID")] + MalformedSaid, + #[error("an Ed25519 signature is 64 bytes; got {0}")] + MalformedSignature(usize), + #[error("not an Ed25519 public key")] + MalformedVerifyingKey, + #[error("the signature does not verify")] + SignatureInvalid, +} + +impl SaidSignatureError { + /// The error class as the conformance pack names it. + pub fn class(&self) -> &'static str { + match self { + SaidSignatureError::MalformedSaid => "malformed_said", + SaidSignatureError::MalformedSignature(_) => "malformed_signature", + SaidSignatureError::MalformedVerifyingKey => "malformed_verifying_key", + SaidSignatureError::SignatureInvalid => "signature_invalid", + } + } +} + +/// Verify `signature` over the SAID whose identity-spec text form is +/// `said_text`, under `verifying_key`. +/// +/// Verification is RFC 8032 Ed25519, strict: a signature scalar `S` outside +/// `[0, L)` is refused (RFC 8032 §5.1.7), and so are small-order keys and +/// commitments, which plain verification would let a weak key satisfy. +/// Structural problems are reported before any cryptography runs, so a short +/// signature is `malformed_signature` rather than a verification failure. +pub fn verify_said_signature( + said_text: &str, + signature: &[u8], + verifying_key: &[u8], +) -> Result<(), SaidSignatureError> { + let said = Said::from_said_text(said_text).map_err(|_| SaidSignatureError::MalformedSaid)?; + let sig: [u8; 64] = signature + .try_into() + .map_err(|_| SaidSignatureError::MalformedSignature(signature.len()))?; + let key: [u8; 32] = verifying_key + .try_into() + .map_err(|_| SaidSignatureError::MalformedVerifyingKey)?; + let key = + VerifyingKey::from_bytes(&key).map_err(|_| SaidSignatureError::MalformedVerifyingKey)?; + key.verify_strict(said.as_bytes(), &Signature::from_bytes(&sig)) + .map_err(|_| SaidSignatureError::SignatureInvalid) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The Ed25519 group order L, little-endian. + const L: [u8; 32] = [ + 0xed, 0xd3, 0xf5, 0x5c, 0x1a, 0x63, 0x12, 0x58, 0xd6, 0x9c, 0xf7, 0xa2, 0xde, 0xf9, 0xde, + 0x14, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x10, + ]; + + fn add_l(s: &[u8]) -> [u8; 32] { + let mut out = [0u8; 32]; + let mut carry = 0u16; + for i in 0..32 { + let v = u16::from(s[i]) + u16::from(L[i]) + carry; + out[i] = v as u8; + carry = v >> 8; + } + assert_eq!(carry, 0); + out + } + + #[test] + fn a_signature_over_a_said_verifies_and_nothing_else_does() { + let key = SigningKey::from_bytes(&[9u8; 32]); + let said = Said::hash(b"one"); + let text = said.to_said_text(); + let vk = key.verifying_key().to_bytes(); + let sig = sign_said(&said, &key).to_bytes(); + + assert_eq!(verify_said_signature(&text, &sig, &vk), Ok(())); + let other = Said::hash(b"two").to_said_text(); + assert_eq!( + verify_said_signature(&other, &sig, &vk), + Err(SaidSignatureError::SignatureInvalid) + ); + let stranger = SigningKey::from_bytes(&[8u8; 32]) + .verifying_key() + .to_bytes(); + assert_eq!( + verify_said_signature(&text, &sig, &stranger), + Err(SaidSignatureError::SignatureInvalid) + ); + assert_eq!( + verify_said_signature(&text, &sig[..63], &vk), + Err(SaidSignatureError::MalformedSignature(63)) + ); + assert_eq!( + verify_said_signature("Enope", &sig, &vk), + Err(SaidSignatureError::MalformedSaid) + ); + } + + #[test] + fn the_same_scalar_spelled_s_plus_l_is_refused() { + let key = SigningKey::from_bytes(&[3u8; 32]); + let said = Said::hash(b"malleable"); + let mut sig = sign_said(&said, &key).to_bytes(); + let s_plus_l = add_l(&sig[32..]); + sig[32..].copy_from_slice(&s_plus_l); + assert_eq!( + verify_said_signature(&said.to_said_text(), &sig, &key.verifying_key().to_bytes()), + Err(SaidSignatureError::SignatureInvalid) + ); + } +} -- 2.54.0 (Apple Git-157) From 49b719b90b482566560886805b6dcf4fec0a2f5b Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Wed, 30 Sep 2026 20:30:04 -0400 Subject: [PATCH 05/12] smart-byte-lockstep: the commit rule and dissent log as a pure function MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `tally` takes the state hashes a frame's nodes reported and returns what the treatise says follows: the floor(2n/3)+1 threshold (§13.1), each node's frame outcome — clean commit, commit with dissent, minority divergence, cluster split (§13.3) — and the dissent-log entry the committing nodes write (§14.1). A split frame records no dissent: dissent is divergence from a committed hash (§14.2). Missing reports still count against n; duplicate reports are refused. `Cluster::step` now goes through it, so the simulation and the conformance vectors share one implementation of the rule. It also keeps the dissent log the treatise calls the substrate's real misbehaviour signal (§13.4): a node that lies about its hash is now named on every frame it lied, where before the commit simply succeeded and the lie left no trace. Co-Authored-By: Claude Opus 5.5 --- .../crates/smart-byte-lockstep/src/lib.rs | 306 ++++++++++++++++-- 1 file changed, 288 insertions(+), 18 deletions(-) diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-lockstep/src/lib.rs b/smart-byte/smart-byte-rs/crates/smart-byte-lockstep/src/lib.rs index 8d186220f..15c0f34ea 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-lockstep/src/lib.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-lockstep/src/lib.rs @@ -9,8 +9,14 @@ //! Faulty nodes are modeled as nodes that *flip a bit* in their state //! before voting; the cluster still commits if at least `floor(2n/3)+1` //! honest nodes agree. +//! +//! The commit decision itself is [`tally`]: a pure function from the state +//! hashes a frame's nodes reported to the treatise's outcomes — the +//! supermajority threshold (§13.1), the four frame outcomes (§13.3) and the +//! dissent-log entry a committing node writes (§14.1–§14.2). [`Cluster::step`] +//! uses it, and the conformance pack's lockstep vectors are checked against it. -use std::collections::{HashMap, HashSet}; +use std::collections::{BTreeMap, HashSet}; use serde::{Deserialize, Serialize}; use smart_byte_core::{Blake3Hash, Said, ownership::Transition}; @@ -28,6 +34,158 @@ pub enum LockstepError { }, } +/// The supermajority a cluster of `n` nodes needs to commit a frame: +/// `floor(2n/3) + 1` (treatise §13.1). +pub fn supermajority_threshold(n: usize) -> usize { + 2 * n / 3 + 1 +} + +/// The four outcomes a frame can reach, as one node sees them (treatise §13.3). +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum FrameOutcome { + /// Supermajority on this node's hash, and every report agreed. + CleanCommit, + /// Supermajority on this node's hash, but some peers reported another. + CommitWithDissent, + /// Supermajority on a hash that is not this node's: it halts. + MinorityDivergence, + /// No hash reached the supermajority: every node halts. + ClusterSplit, +} + +impl FrameOutcome { + /// The outcome as the conformance pack names it. + pub fn as_str(&self) -> &'static str { + match self { + FrameOutcome::CleanCommit => "clean_commit", + FrameOutcome::CommitWithDissent => "commit_with_dissent", + FrameOutcome::MinorityDivergence => "minority_divergence", + FrameOutcome::ClusterSplit => "cluster_split", + } + } +} + +/// One dissent-log entry (treatise §14.1): the frame, the peers whose +/// reported hash differed from the supermajority hash, the hash the logging +/// node computed, and the supermajority hash. +/// +/// Only a node that commits writes one, so `local_hash` always equals +/// `supermajority_hash` — §14.1 says so, and keeping both fields makes an +/// entry from a node that broke that rule detectable. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct DissentEntry

{ + pub frame: u64, + /// In ascending order, so every honest node writes the same entry (§14.2). + pub dissenting_peers: Vec

, + pub local_hash: Blake3Hash, + pub supermajority_hash: Blake3Hash, +} + +/// The state-hash reports for one frame, tallied. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Tally

{ + /// Nodes in the cluster — the `n` of the threshold, which counts nodes + /// that did not report. + pub cluster_size: usize, + /// Votes for the most-reported hash. + pub leading_votes: usize, + /// The hash reported by at least [`supermajority_threshold`] nodes, if + /// any. At most one hash can be: the threshold exceeds half the cluster. + pub supermajority_hash: Option, + /// Peers whose report differs from the supermajority hash, ascending. + /// Empty when there is no supermajority: dissent is divergence *from* a + /// committed hash, so a split frame records none (§14.2). + pub dissenters: Vec

, +} + +/// Why a set of reports could not be tallied. +#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)] +pub enum TallyError { + #[error("cluster is empty")] + EmptyCluster, + #[error("{reports} state-hash reports for a cluster of {cluster_size}")] + TooManyReports { reports: usize, cluster_size: usize }, + #[error("a peer reported twice for one frame")] + DuplicateReport, +} + +/// Tally one frame's state-hash reports for a cluster of `cluster_size`. +pub fn tally( + cluster_size: usize, + reports: &[(P, Blake3Hash)], +) -> Result, TallyError> { + if cluster_size == 0 { + return Err(TallyError::EmptyCluster); + } + if reports.len() > cluster_size { + return Err(TallyError::TooManyReports { + reports: reports.len(), + cluster_size, + }); + } + let mut by_peer: BTreeMap<&P, &Blake3Hash> = BTreeMap::new(); + for (peer, hash) in reports { + if by_peer.insert(peer, hash).is_some() { + return Err(TallyError::DuplicateReport); + } + } + let mut votes: BTreeMap<&Blake3Hash, usize> = BTreeMap::new(); + for hash in by_peer.values() { + *votes.entry(*hash).or_default() += 1; + } + let leading_votes = votes.values().copied().max().unwrap_or(0); + let threshold = supermajority_threshold(cluster_size); + let supermajority_hash = votes + .iter() + .find(|(_, count)| **count >= threshold) + .map(|(hash, _)| **hash); + let dissenters = match supermajority_hash { + Some(winner) => by_peer + .iter() + .filter(|(_, hash)| ***hash != winner) + .map(|(peer, _)| (*peer).clone()) + .collect(), + None => Vec::new(), + }; + Ok(Tally { + cluster_size, + leading_votes, + supermajority_hash, + dissenters, + }) +} + +impl Tally

{ + /// Whether the frame commits. + pub fn committed(&self) -> bool { + self.supermajority_hash.is_some() + } + + /// The §13.3 outcome for a node whose own post-frame hash is `local`. + pub fn outcome(&self, local: &Blake3Hash) -> FrameOutcome { + match &self.supermajority_hash { + None => FrameOutcome::ClusterSplit, + Some(winner) if winner != local => FrameOutcome::MinorityDivergence, + Some(_) if self.dissenters.is_empty() => FrameOutcome::CleanCommit, + Some(_) => FrameOutcome::CommitWithDissent, + } + } + + /// The dissent-log entry a node whose own hash is `local` writes for + /// `frame`: present only for [`FrameOutcome::CommitWithDissent`]. + pub fn dissent_entry(&self, frame: u64, local: &Blake3Hash) -> Option> { + match self.outcome(local) { + FrameOutcome::CommitWithDissent => Some(DissentEntry { + frame, + dissenting_peers: self.dissenters.clone(), + local_hash: *local, + supermajority_hash: self.supermajority_hash?, + }), + _ => None, + } + } +} + /// A single ordered batch of state transitions. #[derive(Clone, Debug, Serialize, Deserialize)] pub struct Frame { @@ -94,12 +252,17 @@ pub struct FrameCommit { pub frame: u64, pub state_hash: Blake3Hash, pub committed_by: HashSet, + /// Nodes whose announced hash differed from `state_hash` (§14.1). + pub dissenters: Vec, } /// A federated cluster of [`Node`]s running deterministic lockstep. pub struct Cluster { pub nodes: Vec, pub frame: u64, + /// The dissent log the committing nodes keep (treatise §14). Every honest + /// node writes the same entry for a frame, so the simulation keeps one. + pub dissent_log: Vec>, } impl Cluster { @@ -111,7 +274,11 @@ impl Cluster { Node::new(id) }) .collect(); - Self { nodes, frame: 0 } + Self { + nodes, + frame: 0, + dissent_log: Vec::new(), + } } /// Build a cluster of `n` nodes where `faulty_count` of them are @@ -136,29 +303,32 @@ impl Cluster { ordered_transitions: transitions, }; let total = self.nodes.len(); - let mut tally: HashMap> = HashMap::new(); - for node in &mut self.nodes { - let vote = node.step(&frame); - tally.entry(vote).or_default().insert(node.id); - } - // Pick the hash with the largest backing set. - let (best_hash, voters) = tally - .into_iter() - .max_by_key(|(_, voters)| voters.len()) - .expect("non-empty cluster has at least one vote"); - // Strict > 2/3 (BFT supermajority). - let needed = (2 * total) / 3 + 1; - if voters.len() < needed { + let reports: Vec<(Said, Blake3Hash)> = self + .nodes + .iter_mut() + .map(|node| (node.id, node.step(&frame))) + .collect(); + let result = tally(total, &reports).expect("one report per node in a non-empty cluster"); + let Some(state_hash) = result.supermajority_hash else { return Err(LockstepError::NoSupermajority { frame: frame.frame_number, - best: voters.len(), + best: result.leading_votes, total, }); + }; + let committed_by = reports + .iter() + .filter(|(_, hash)| *hash == state_hash) + .map(|(id, _)| *id) + .collect(); + if let Some(entry) = result.dissent_entry(frame.frame_number, &state_hash) { + self.dissent_log.push(entry); } let commit = FrameCommit { frame: frame.frame_number, - state_hash: best_hash, - committed_by: voters, + state_hash, + committed_by, + dissenters: result.dissenters, }; self.frame += 1; Ok(commit) @@ -203,11 +373,111 @@ mod tests { #[test] fn one_faulty_node_still_commits() { let mut cluster = Cluster::new_with_faulty(4, 1); + let liar = cluster.nodes[3].id; // 3-of-4 honest nodes is exactly the floor(2*4/3)+1 = 3 threshold. for f in 0..10 { let commit = cluster.step(synthetic_transitions(f, 3)).unwrap(); assert_eq!(commit.committed_by.len(), 3); + assert_eq!(commit.dissenters, vec![liar]); } + // §14: the liar is caught by the dissent log on every frame it lied. + assert_eq!(cluster.dissent_log.len(), 10); + assert!( + cluster + .dissent_log + .iter() + .all(|e| e.dissenting_peers == vec![liar] && e.local_hash == e.supermajority_hash) + ); + } + + #[test] + fn an_honest_cluster_logs_no_dissent() { + let mut cluster = Cluster::new_honest(4); + for f in 0..3 { + assert!( + cluster + .step(synthetic_transitions(f, 2)) + .unwrap() + .dissenters + .is_empty() + ); + } + assert!(cluster.dissent_log.is_empty()); + } + + #[test] + fn the_threshold_table_of_section_13_1() { + for (n, want) in [ + (2, 2), + (3, 3), + (4, 3), + (5, 4), + (6, 5), + (7, 5), + (10, 7), + (16, 11), + (32, 22), + (64, 43), + ] { + assert_eq!(supermajority_threshold(n), want, "n = {n}"); + } + } + + #[test] + fn tally_gives_each_node_its_own_outcome() { + let (a, b) = ([0xaa; 32], [0xbb; 32]); + let t = tally(4, &[(1, a), (2, a), (3, a), (4, b)]).unwrap(); + assert_eq!(t.supermajority_hash, Some(a)); + assert_eq!(t.dissenters, vec![4]); + assert_eq!(t.outcome(&a), FrameOutcome::CommitWithDissent); + assert_eq!(t.outcome(&b), FrameOutcome::MinorityDivergence); + assert!( + t.dissent_entry(9, &b).is_none(), + "a halted node writes no entry" + ); + let e = t.dissent_entry(9, &a).unwrap(); + assert_eq!( + (e.frame, e.dissenting_peers, e.supermajority_hash), + (9, vec![4], a) + ); + } + + #[test] + fn a_split_frame_records_no_dissent() { + // §13.2: at n = 3 one liar leaves 2 of the 3 votes the frame needs. + let (a, b) = ([1; 32], [2; 32]); + let t = tally(3, &[(1, a), (2, a), (3, b)]).unwrap(); + assert_eq!(t.supermajority_hash, None); + assert_eq!(t.leading_votes, 2); + assert!(t.dissenters.is_empty()); + assert_eq!(t.outcome(&a), FrameOutcome::ClusterSplit); + assert!(t.dissent_entry(1, &a).is_none()); + } + + #[test] + fn missing_reports_count_against_the_threshold() { + // n stays the cluster size: 3 agreeing reports commit at n = 4 but + // not at n = 5, whoever failed to report. + let a = [7; 32]; + assert!(tally(4, &[(1, a), (2, a), (3, a)]).unwrap().committed()); + assert!(!tally(5, &[(1, a), (2, a), (3, a)]).unwrap().committed()); + } + + #[test] + fn malformed_report_sets_are_refused() { + let a = [7; 32]; + assert_eq!(tally::(0, &[]), Err(TallyError::EmptyCluster)); + assert_eq!( + tally(1, &[(1, a), (2, a)]), + Err(TallyError::TooManyReports { + reports: 2, + cluster_size: 1 + }) + ); + assert_eq!( + tally(3, &[(1, a), (1, a)]), + Err(TallyError::DuplicateReport) + ); } #[test] -- 2.54.0 (Apple Git-157) From a4b899a48cf6e780f265c576566b2a6c004b16e4 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Wed, 30 Sep 2026 20:30:04 -0400 Subject: [PATCH 06/12] smart-byte-libp2p: check what arrives against the SAID it claims MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit No receive path verified anything. `fetch_by_said` returned whatever envelope the peer sent, so a peer could answer a request for one envelope with another. The three gossip receive paths decoded envelopes and delivered them under whatever id they stated, so an envelope whose content did not hash to its id propagated exactly like a genuine one. Every gossip path now goes through `accept_envelope` (decode, then the content must hash to its stated id — treatise §10.2 step 1). A fetch answer must state the requested SAID and hash to it, and a different envelope is refused with `Error::Substituted`. Unit tests cover a forged body under a genuine id, a different envelope, and a relabelled one. This uses the envelope's existing identity derivation; moving envelope ids to the identity spec's SAIDs is a separate, wider change. Co-Authored-By: Claude Opus 5.5 --- .../crates/smart-byte-libp2p/src/error.rs | 11 +++ .../crates/smart-byte-libp2p/src/event.rs | 4 +- .../crates/smart-byte-libp2p/src/transport.rs | 95 ++++++++++++++++++- 3 files changed, 104 insertions(+), 6 deletions(-) diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/error.rs b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/error.rs index f49c352a4..0a8b717bb 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/error.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/error.rs @@ -41,6 +41,17 @@ pub enum Error { reason: String, }, + /// A peer answered a fetch for one SAID with a different envelope. + #[error("{peer} answered a fetch for {requested} with {served}")] + Substituted { + /// Peer that answered. + peer: PeerId, + /// The SAID that was asked for. + requested: smart_byte_core::Said, + /// The id the answer stated. + served: smart_byte_core::Said, + }, + /// The control channel for driving the swarm was closed. #[error("control channel closed")] ControlClosed, diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/event.rs b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/event.rs index 0316dc52e..d8bcb6721 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/event.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/event.rs @@ -65,14 +65,14 @@ fn translate(ev: SwarmEvent) -> Option { message, .. }, - )) => match Envelope::from_cbor(&message.data) { + )) => match crate::transport::accept_envelope(&message.data) { Ok(envelope) => Some(NodeEvent::EnvelopeReceived { envelope: Box::new(envelope), from: propagation_source, topic: message.topic.into_string(), }), Err(e) => { - tracing::warn!(error = %e, "received malformed envelope on gossipsub"); + tracing::warn!(error = %e, "dropped a malformed or mis-addressed envelope from gossipsub"); None } }, diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/transport.rs b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/transport.rs index 701e7840d..549b48a45 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/transport.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/transport.rs @@ -39,6 +39,35 @@ use crate::codec::{EnvelopeRequest, EnvelopeResponse}; use crate::error::{Error, Result}; use crate::node::Libp2pNode; +/// Decode an envelope received from the network, accepting it only if its +/// content hashes to the id it states (treatise §10.2 step 1). +/// +/// Every gossip receive path goes through here. Before it did, none of them +/// checked, so an envelope stating any id was delivered under that id: a +/// substituted envelope propagated exactly like the real one. +pub(crate) fn accept_envelope( + bytes: &[u8], +) -> std::result::Result { + let env = Envelope::from_cbor(bytes)?; + env.verify_said()?; + Ok(env) +} + +/// Accept a fetch's answer only if it is the envelope that was asked for: it +/// must state the requested SAID, and its content must hash to it. Otherwise a +/// peer could answer any request with any envelope it liked. +fn check_served(peer: PeerId, requested: Said, envelope: Envelope) -> Result { + if envelope.id != requested { + return Err(Error::Substituted { + peer, + requested, + served: envelope.id, + }); + } + envelope.verify_said()?; + Ok(envelope) +} + /// Publish an envelope on a gossipsub topic. Returns the `MessageId` /// assigned by gossipsub. pub async fn publish_envelope( @@ -103,7 +132,7 @@ pub async fn subscribe<'a>( }, )) = ev && message.topic.as_str() == topic_owned - && let Ok(env) = Envelope::from_cbor(&message.data) + && let Ok(env) = accept_envelope(&message.data) { return Some(( (env, propagation_source), @@ -139,8 +168,7 @@ pub async fn next_envelope_on( .. }, ))) if message.topic.as_str() == topic => { - let env = Envelope::from_cbor(&message.data) - .map_err(Error::Envelope)?; + let env = accept_envelope(&message.data).map_err(Error::Envelope)?; return Ok((env, propagation_source)); } Ok(_) => {} @@ -207,7 +235,7 @@ async fn direct_fetch(node: &mut Libp2pNode, peer: PeerId, said: Said) -> Result match tokio::time::timeout(remaining, node.swarm.select_next_some()).await { Ok(ev) => { if let Some(envelope) = handle_event_for_response(ev, request_id, peer)? { - return Ok(envelope); + return check_served(peer, said, envelope); } } Err(_) => { @@ -302,3 +330,62 @@ async fn find_provider(node: &mut Libp2pNode, said: Said) -> Result { } } } + +#[cfg(test)] +mod tests { + use super::*; + use chrono::TimeZone; + use smart_byte_core::{Cargo, EnvelopeError, JouleCost, OwnershipChain, Provenance}; + + fn envelope(seed: &[u8]) -> Envelope { + let issued_at = chrono::Utc.with_ymd_and_hms(2026, 9, 30, 0, 0, 0).unwrap(); + Envelope::new( + Provenance::new(Said::hash(seed), issued_at, vec![]), + OwnershipChain::empty(), + Cargo::Usd { minor: 100 }, + JouleCost::measured(1), + ) + .unwrap() + } + + #[test] + fn a_received_envelope_must_hash_to_its_own_id() { + let genuine = envelope(b"issuer"); + assert_eq!( + accept_envelope(&genuine.to_cbor().unwrap()).unwrap(), + genuine + ); + + // Same id, different content: refused rather than delivered under it. + let mut forged = genuine.clone(); + forged.cargo = Cargo::Usd { minor: 1_000_000 }; + assert!(matches!( + accept_envelope(&forged.to_cbor().unwrap()), + Err(EnvelopeError::SaidMismatch { .. }) + )); + } + + #[test] + fn a_fetch_answered_with_another_envelope_is_refused() { + let peer = PeerId::random(); + let wanted = envelope(b"wanted"); + let other = envelope(b"other"); + + assert_eq!( + check_served(peer, wanted.id, wanted.clone()).unwrap(), + wanted + ); + assert!(matches!( + check_served(peer, wanted.id, other.clone()), + Err(Error::Substituted { requested, served, .. }) + if requested == wanted.id && served == other.id + )); + // Stating the requested id is not enough: the content must hash to it. + let mut relabelled = other; + relabelled.id = wanted.id; + assert!(matches!( + check_served(peer, wanted.id, relabelled), + Err(Error::Envelope(EnvelopeError::SaidMismatch { .. })) + )); + } +} -- 2.54.0 (Apple Git-157) From becbfa8466a0d4ab897e49c94c3c04b2c5ea0a6e Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Wed, 30 Sep 2026 20:30:04 -0400 Subject: [PATCH 07/12] smart-byte: generate the conformance pack from the reference implementation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The v1 pack was filler. Its SAIDs were runs of one repeated character, every `canonical_cbor_hex` was the same 7-byte stub, and its signatures were RFC 8032 test signatures over the empty message. All five canonicalisation encodings were wrong; two were odd-length hex that could not be decoded at all. The gate only ran `jq` over it, and the spec said implementers MUST pass it. Now every value is computed, and checked twice: - New crate `smart-byte-conformance`. It is the `gen-conformance-vectors` / `verify-conformance-vectors` pair that regeneration.md always promised and that never existed. The generator computes every expected value with smart-byte-core and smart-byte-lockstep, and fails if a negative vector is refused for any reason other than the one it names. Its `cargo test` (run by standards-build on every push) checks: * every published vector verifies; * the published files equal the generator's output byte for byte; * a defect planted in each file is caught on the vector it was planted in; * identity spec §2.4 prints the worked example's real values. - `conformance/crosscheck/verify_v1.py`: a second implementation, in Python, stdlib only, with BLAKE3 and Ed25519 written from their specs and checked against known answers first. It reproduces all 47 vectors, and the conformance gate now runs it. The pack now tests what the spec says: - canonicalisation: each §2.3 rule, accepted and refused (19 vectors). - SAIDs: the suite §2.5 names, which the pack never contained — the empty-map case, the one-field case, the worked example, five envelope-shaped objects, and two adversarial non-canonical encodings (one aimed at re-encoding verifiers, one at verifiers that hash bytes as received). - signatures: over real SAIDs, plus a non-canonical S. - lockstep: §13.3 outcomes and §14.1 dissent entries. - gossip: content-addressed intake. Vectors that contradicted the spec are corrected: - canon-004 encoded floats, which §2.3 forbids; - lockstep-003 logged dissent on a frame that never committed; - gossip-002 settled a "collision" by arrival order; - gossip-003 asserted backfill limits no spec defines. regeneration.md records each change and why the draft pack was regenerated in place, not by supersession. Spec: §2.4 printed an invented digest and SAID, and miscounted the encoding as 62 bytes (it is 61); it now prints the real values. §2.5 names the vectors. §6 pointed at modules that did not exist, and now says where each part lives — including that the key-event log does not derive §2 SAIDs yet. Replaces this branch's placeholder pins in smart-byte-core/tests/pack.rs, which were written to fail once the pack was regenerated, with the assertions they asked for. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/conformance.yml | 25 + smart-byte/conformance/README.md | 97 +- smart-byte/conformance/VERSIONS.md | 2 +- .../conformance/crosscheck/verify_v1.py | 611 ++++++++++++ .../v1/canonicalisation_vectors.json | 154 ++- smart-byte/conformance/v1/gossip_vectors.json | 201 +++- .../conformance/v1/lockstep_vectors.json | 172 ++-- smart-byte/conformance/v1/regeneration.md | 272 ++++-- smart-byte/conformance/v1/said_vectors.json | 195 +++- .../conformance/v1/signature_vectors.json | 62 +- smart-byte/smart-byte-rs/Cargo.lock | 12 +- smart-byte/smart-byte-rs/Cargo.toml | 1 + .../crates/smart-byte-conformance/Cargo.toml | 23 + .../src/bin/gen-conformance-vectors.rs | 67 ++ .../src/bin/verify-conformance-vectors.rs | 40 + .../smart-byte-conformance/src/gossip.rs | 71 ++ .../crates/smart-byte-conformance/src/json.rs | 252 +++++ .../crates/smart-byte-conformance/src/lib.rs | 20 + .../crates/smart-byte-conformance/src/pack.rs | 923 ++++++++++++++++++ .../smart-byte-conformance/src/verify.rs | 473 +++++++++ .../smart-byte-conformance/tests/pack.rs | 292 ++++++ .../crates/smart-byte-core/Cargo.toml | 4 - .../crates/smart-byte-core/tests/pack.rs | 262 ----- smart-byte/spec/identity_and_key_rotation.md | 30 +- 24 files changed, 3666 insertions(+), 595 deletions(-) create mode 100644 smart-byte/conformance/crosscheck/verify_v1.py create mode 100644 smart-byte/smart-byte-rs/crates/smart-byte-conformance/Cargo.toml create mode 100644 smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/bin/gen-conformance-vectors.rs create mode 100644 smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/bin/verify-conformance-vectors.rs create mode 100644 smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/gossip.rs create mode 100644 smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/json.rs create mode 100644 smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/lib.rs create mode 100644 smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/pack.rs create mode 100644 smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/verify.rs create mode 100644 smart-byte/smart-byte-rs/crates/smart-byte-conformance/tests/pack.rs delete mode 100644 smart-byte/smart-byte-rs/crates/smart-byte-core/tests/pack.rs diff --git a/.github/workflows/conformance.yml b/.github/workflows/conformance.yml index c84053057..925bcc32a 100644 --- a/.github/workflows/conformance.yml +++ b/.github/workflows/conformance.yml @@ -84,6 +84,31 @@ jobs: exit 1 fi + # The pack's values come from the Rust reference implementation, and + # standards-build checks them against it under `cargo test`. This job checks + # them against a second implementation, written separately in another + # language, so a mistake shared by the generator and its own verifier still + # fails here. Standard library only: nothing to install beyond python3. + crosscheck: + name: Second implementation reproduces every vector + runs-on: linux + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Ensure python3 is available + shell: bash + run: | + if ! command -v python3 >/dev/null 2>&1; then + SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo" + $SUDO apt-get update -qq + $SUDO apt-get install -y -qq python3 + fi + python3 --version + + - name: Cross-check v1 + run: python3 smart-byte/conformance/crosscheck/verify_v1.py smart-byte/conformance/v1 + spec-vector-coupling: name: Spec changes require vector-pack updates runs-on: linux diff --git a/smart-byte/conformance/README.md b/smart-byte/conformance/README.md index 9ad294e42..8f9f5c438 100644 --- a/smart-byte/conformance/README.md +++ b/smart-byte/conformance/README.md @@ -17,21 +17,24 @@ contain no `$ref` pointers, no external URLs, and no runtime dependencies. ## What is in the pack Each spec version has its own subdirectory (`v1/`, `v2/`, ...). Today only -`v1/` exists; it corresponds to the spec text in -`../spec/treatise_v1_parts_I_II_III_combined.md`. +`v1/` exists; it corresponds to the spec text in `../spec/` +(`treatise_v1_parts_I_II_III_combined.md` and +`identity_and_key_rotation.md`). A pack for a given version contains five JSON files plus `regeneration.md`: -| File | Tests | -| -------------------------------- | -------------------------------------------------------- | -| `canonicalisation_vectors.json` | CBOR canonical-encoding rules (§8 wire format) | -| `said_vectors.json` | SAID computation: BLAKE3-256 over canonical CBOR (§8.2) | -| `signature_vectors.json` | Ed25519 sign + verify over SAIDs (§8.3, §8.4) | -| `lockstep_vectors.json` | BFT supermajority commit logic (§9, §13) | -| `gossip_vectors.json` | Gossip propagation, conflict detection, ordering (§17) | +| File | Tests | +| -------------------------------- | --------------------------------------------------------------------- | +| `canonicalisation_vectors.json` | Canonical CBOR: each encoding rule, accepted and refused (identity §2.3) | +| `said_vectors.json` | SAID derivation and verification (identity §2.1–§2.5; treatise §8.2) | +| `signature_vectors.json` | Ed25519 signatures over SAIDs (treatise §8.3, §8.4; RFC 8032) | +| `lockstep_vectors.json` | Supermajority commit, frame outcomes, dissent log (treatise §13, §14) | +| `gossip_vectors.json` | Content-addressed intake of gossiped objects (treatise §10.2, §11.4, §21.4) | -`regeneration.md` documents how to regenerate the vectors from the Rust -reference implementation (`smart-byte-rs`, landing alongside spec v1.0). +Every value in the pack is generated by the Rust reference implementation +(`../smart-byte-rs`, crate `smart-byte-conformance`) and reproduced by an +independent second implementation, `crosscheck/verify_v1.py`. +`regeneration.md` documents both, and records every change to the pack. ## How to consume the vectors @@ -60,6 +63,71 @@ The conventional flow for an implementation under test (IUT): There is no scoring, no "mostly conforms" status, and no waiver process. A vector pack is a hard gate. +### Byte strings in JSON + +JSON has no byte-string type, so an `input` writes a CBOR byte string as an +object whose only member is `$bytes`, holding lowercase hex: +`{"$bytes": "00ff"}`. Every other JSON value maps to CBOR directly: object → +map with text keys, array → array, string → text string, integer → integer, +`true` / `false` / `null` → the simple values. The pack contains no JSON number +with a fraction or exponent, because floats have no canonical form. + +Where a vector supplies raw bytes instead (`*_cbor_hex`, `input_cbor_hex`), +they are lowercase hex. + +### Fields, file by file + +- **`canonicalisation_vectors.json`** — a positive vector has `input` and + `expected_cbor_hex`; a negative one has `input_cbor_hex`, which a strict + decoder must refuse with `error`. Refusing means not decoding: an IUT must + never re-encode non-canonical input into canonical form and accept it. +- **`said_vectors.json`** — `said_field` names the member that carries the + SAID (`d`). A positive vector's `input` holds the 53-`#` placeholder there. + `canonical_cbor_hex` is the canonical encoding of `input` (the bytes that + are hashed), `blake3_hex` is their BLAKE3-256, `expected_said` is that + digest written `E` + lowercase unpadded base32, and `said_cbor_hex` is the + object with its SAID in place, which must verify. A negative vector's + `input_cbor_hex` must fail SAID verification with `error`. The adversarial + vectors also give, for illustration, the SAIDs a lenient verifier would + compute (`said_of_received_bytes`, `said_after_reencoding`). +- **`signature_vectors.json`** — `signed_message_hex` is the message: the 32 + digest bytes `envelope_said` spells, not its text. `verifying_key` is the + key `signing_key_seed` derives. Verify `signature` under `verifying_key`: + the result must be `expected_verify`, failing with `error` when false. +- **`lockstep_vectors.json`** — `node_hashes` are the opaque 32-byte state + hashes each node reported, compared only for equality, and `node_count` is + the cluster size n. The IUT computes `expected_threshold` (floor(2n/3)+1), + `expected_commit`, `expected_state_hash`, each node's `expected_outcomes` + (`clean_commit`, `commit_with_dissent`, `minority_divergence` or + `cluster_split`) and the dissent-log entries the committing nodes write + (`expected_dissent`). +- **`gossip_vectors.json`** — `deliveries` arrive at `receiver` in order, each + with the SAID it was announced or requested under (`claimed_said`) and its + bytes (`cbor_hex`). Each result is `accepted` or `duplicate` with the SAID, + or `rejected` with an error class. `expected_held` lists the SAIDs the + receiver holds afterwards, in the order it accepted them. + +### Error classes + +| Class | Meaning | +| ------------------------- | ---------------------------------------------------------- | +| `non_shortest_integer` | an integer not written with its shortest head | +| `non_shortest_length` | a string, array or map length not written with its shortest head | +| `unsorted_map_keys` | map keys not in the bytewise order of their encodings | +| `duplicate_map_key` | a key appears twice in one map | +| `float_forbidden` | a floating-point value | +| `tag_forbidden` | a CBOR tag | +| `indefinite_length` | an indefinite-length item | +| `invalid_utf8` | a text string that is not UTF-8 | +| `trailing_bytes` | bytes after the one top-level item | +| `said_mismatch` | content that does not hash to the SAID it states or arrived under | +| `signature_invalid` | a signature that does not verify, including a non-canonical S | +| `malformed_signature` | a signature that is not 64 bytes | + +The reference implementation also reports `truncated`, `malformed`, +`too_deep`, `not_a_map`, `missing_said_field`, `malformed_said` and +`malformed_verifying_key` for inputs no vector exercises. + ## Versioning rule Vectors are versioned **per spec version**, not per implementation, not per @@ -84,10 +152,17 @@ Every PR that touches `smart-byte/spec/` or `smart-byte/conformance/` must pass the gate. The gate validates: - Every JSON file in `v*/` parses as valid JSON. +- The second implementation, `crosscheck/verify_v1.py`, agrees with every + vector. - Every spec change is accompanied by a vector-pack update (a PR that modifies `smart-byte/spec/` without modifying `smart-byte/conformance/` is rejected at the gate). +Separately, `standards-build` runs `cargo test` in `smart-byte-rs` on every +push. That checks every vector against the reference implementation, requires +the published files to equal the generator's output byte for byte, and +requires identity spec §2.4 to print the worked example's real values. + The gate's purpose is procedural: a spec change without a vector change is either a clarifying edit (which should declare itself by also touching the vector pack's `regeneration.md` to record the editorial reason) or a diff --git a/smart-byte/conformance/VERSIONS.md b/smart-byte/conformance/VERSIONS.md index 32b5c1269..d323ddc2e 100644 --- a/smart-byte/conformance/VERSIONS.md +++ b/smart-byte/conformance/VERSIONS.md @@ -7,7 +7,7 @@ Smart Byte spec versions. | Vector pack | Spec version | Spec source | Status | Notes | | ----------- | ------------ | ------------------------------------------------------------- | -------- | --------------------------------------------------------------------- | -| `v1/` | v1.0-draft | `../spec/treatise_v1_parts_I_II_III_combined.md` (Parts I-III) | DRAFT | Reference vectors; expected to be regenerated when `smart-byte-rs` lands. | +| `v1/` | v1.0-draft | `../spec/treatise_v1_parts_I_II_III_combined.md` (Parts I-III), `../spec/identity_and_key_rotation.md` | DRAFT | Generated by the reference implementation (`smart-byte-conformance`) and reproduced by `crosscheck/verify_v1.py`; regenerated in place on 2026-09-30, see `v1/regeneration.md`. Supersession (rule 4) applies once v1 freezes. | Future entries are appended below this table when a new spec version is introduced. Older entries are never removed; an implementation that pins to diff --git a/smart-byte/conformance/crosscheck/verify_v1.py b/smart-byte/conformance/crosscheck/verify_v1.py new file mode 100644 index 000000000..ac7c68747 --- /dev/null +++ b/smart-byte/conformance/crosscheck/verify_v1.py @@ -0,0 +1,611 @@ +#!/usr/bin/env python3 +"""Second implementation of the Smart Byte v1 conformance pack. + +The pack in ../v1 is generated by the Rust reference implementation +(smart-byte-rs, crate smart-byte-conformance). This script is a separate +implementation of everything the pack exercises, written in another language +from the spec text rather than ported from the Rust: canonical CBOR and its +strict decoder (identity spec §2.3), SAID derivation and verification (§2.1, +§2.2), Ed25519 (RFC 8032), the lockstep commit rule and dissent log (treatise +§13, §14), and the content-addressed gossip intake the gossip vectors describe. + +It uses the Python standard library only. BLAKE3 and Ed25519 are implemented +here from their specifications, and both are checked against published +known-answer values before any vector is. + + python3 verify_v1.py [PACK_DIR] + +Exits 0 when every vector agrees with this implementation, 1 otherwise. +""" + +import base64 +import hashlib +import json +import pathlib +import sys + +# --------------------------------------------------------------------------- +# BLAKE3 (hash mode, 32-byte output) +# --------------------------------------------------------------------------- + +_M32 = 0xFFFFFFFF +_IV = [0x6A09E667, 0xBB67AE85, 0x3C6EF372, 0xA54FF53A, + 0x510E527F, 0x9B05688C, 0x1F83D9AB, 0x5BE0CD19] +_PERM = [2, 6, 3, 10, 7, 0, 4, 13, 1, 11, 12, 5, 9, 14, 15, 8] +_CHUNK_START, _CHUNK_END, _PARENT, _ROOT = 1, 2, 4, 8 +_BLOCK, _CHUNK = 64, 1024 + + +def _rotr(x, n): + return ((x >> n) | (x << (32 - n))) & _M32 + + +def _g(s, a, b, c, d, x, y): + s[a] = (s[a] + s[b] + x) & _M32 + s[d] = _rotr(s[d] ^ s[a], 16) + s[c] = (s[c] + s[d]) & _M32 + s[b] = _rotr(s[b] ^ s[c], 12) + s[a] = (s[a] + s[b] + y) & _M32 + s[d] = _rotr(s[d] ^ s[a], 8) + s[c] = (s[c] + s[d]) & _M32 + s[b] = _rotr(s[b] ^ s[c], 7) + + +def _compress(cv, block, counter, block_len, flags): + m = [int.from_bytes(block[4 * i:4 * i + 4], "little") for i in range(16)] + s = list(cv) + _IV[:4] + [counter & _M32, (counter >> 32) & _M32, block_len, flags] + for r in range(7): + _g(s, 0, 4, 8, 12, m[0], m[1]) + _g(s, 1, 5, 9, 13, m[2], m[3]) + _g(s, 2, 6, 10, 14, m[4], m[5]) + _g(s, 3, 7, 11, 15, m[6], m[7]) + _g(s, 0, 5, 10, 15, m[8], m[9]) + _g(s, 1, 6, 11, 12, m[10], m[11]) + _g(s, 2, 7, 8, 13, m[12], m[13]) + _g(s, 3, 4, 9, 14, m[14], m[15]) + if r < 6: + m = [m[i] for i in _PERM] + return [s[i] ^ s[i + 8] for i in range(8)] + [s[i + 8] ^ cv[i] for i in range(8)] + + +def _chunk_output(chunk, index): + """(cv, last block, block_len, flags) of a chunk, before its last compression.""" + blocks = [chunk[i:i + _BLOCK] for i in range(0, len(chunk), _BLOCK)] or [b""] + cv = list(_IV) + for i, block in enumerate(blocks): + flags = (_CHUNK_START if i == 0 else 0) | (_CHUNK_END if i == len(blocks) - 1 else 0) + padded = block.ljust(_BLOCK, b"\0") + if i == len(blocks) - 1: + return cv, padded, len(block), flags, index + cv = _compress(cv, padded, index, _BLOCK, flags)[:8] + + +def blake3(data): + """BLAKE3-256 of `data`.""" + chunks = [data[i:i + _CHUNK] for i in range(0, len(data), _CHUNK)] or [b""] + stack = [] + for index, chunk in enumerate(chunks[:-1]): + cv, block, n, flags, ctr = _chunk_output(chunk, index) + node = _compress(cv, block, ctr, n, flags)[:8] + total = index + 1 + while total & 1 == 0: + left = stack.pop() + node = _compress(_IV, _words(left) + _words(node), 0, _BLOCK, _PARENT)[:8] + total >>= 1 + stack.append(node) + out = _chunk_output(chunks[-1], len(chunks) - 1) + while stack: + cv, block, n, flags, ctr = out + right = _compress(cv, block, ctr, n, flags)[:8] + out = (_IV, _words(stack.pop()) + _words(right), _BLOCK, _PARENT, 0) + cv, block, n, flags, ctr = out + words = _compress(cv, block, ctr, n, flags | _ROOT)[:8] + return b"".join(w.to_bytes(4, "little") for w in words) + + +def _words(cv): + return b"".join(w.to_bytes(4, "little") for w in cv) + + +# --------------------------------------------------------------------------- +# Ed25519 (RFC 8032), with the strict checks the pack expects +# --------------------------------------------------------------------------- + +_P = 2 ** 255 - 19 +_L = 2 ** 252 + 27742317777372353535851937790883648493 +_D = -121665 * pow(121666, _P - 2, _P) % _P +_SQRT_M1 = pow(2, (_P - 1) // 4, _P) + + +def _recover_x(y, sign): + if y >= _P: + return None + x2 = (y * y - 1) * pow(_D * y * y + 1, _P - 2, _P) % _P + if x2 == 0: + return None if sign else 0 + x = pow(x2, (_P + 3) // 8, _P) + if (x * x - x2) % _P != 0: + x = x * _SQRT_M1 % _P + if (x * x - x2) % _P != 0: + return None + if x & 1 != sign: + x = _P - x + return x + + +_BY = 4 * pow(5, _P - 2, _P) % _P +_BX = _recover_x(_BY, 0) +_B = (_BX, _BY, 1, _BX * _BY % _P) +_ZERO = (0, 1, 1, 0) + + +def _add(p, q): + a = (p[1] - p[0]) * (q[1] - q[0]) % _P + b = (p[1] + p[0]) * (q[1] + q[0]) % _P + c = 2 * p[3] * q[3] * _D % _P + d = 2 * p[2] * q[2] % _P + e, f, g, h = b - a, d - c, d + c, b + a + return (e * f % _P, g * h % _P, f * g % _P, e * h % _P) + + +def _mul(k, p): + q = _ZERO + while k: + if k & 1: + q = _add(q, p) + p = _add(p, p) + k >>= 1 + return q + + +def _eq(p, q): + return (p[0] * q[2] - q[0] * p[2]) % _P == 0 and (p[1] * q[2] - q[1] * p[2]) % _P == 0 + + +def _encode_point(p): + zi = pow(p[2], _P - 2, _P) + x, y = p[0] * zi % _P, p[1] * zi % _P + return (y | ((x & 1) << 255)).to_bytes(32, "little") + + +def _decode_point(b): + n = int.from_bytes(b, "little") + y, sign = n & ((1 << 255) - 1), n >> 255 + x = _recover_x(y, sign) + return None if x is None else (x, y, 1, x * y % _P) + + +def _small_order(p): + return _eq(_mul(8, p), _ZERO) + + +def _sha512_int(*parts): + return int.from_bytes(hashlib.sha512(b"".join(parts)).digest(), "little") + + +def ed25519_public(seed): + h = hashlib.sha512(seed).digest() + a = int.from_bytes(h[:32], "little") & ((1 << 254) - 8) | (1 << 254) + return _encode_point(_mul(a, _B)) + + +def ed25519_sign(seed, msg): + h = hashlib.sha512(seed).digest() + a = int.from_bytes(h[:32], "little") & ((1 << 254) - 8) | (1 << 254) + pub = _encode_point(_mul(a, _B)) + r = _sha512_int(h[32:], msg) % _L + big_r = _encode_point(_mul(r, _B)) + k = _sha512_int(big_r, pub, msg) % _L + return big_r + ((r + k * a) % _L).to_bytes(32, "little") + + +def ed25519_verify(pub, msg, sig): + """None if valid, else the pack's error class.""" + if len(sig) != 64: + return "malformed_signature" + if len(pub) != 32: + return "malformed_verifying_key" + a = _decode_point(pub) + if a is None: + return "malformed_verifying_key" + r = _decode_point(sig[:32]) + s = int.from_bytes(sig[32:], "little") + if r is None or s >= _L or _small_order(a) or _small_order(r): + return "signature_invalid" + k = _sha512_int(sig[:32], pub, msg) % _L + if not _eq(_mul(s, _B), _add(r, _mul(k, a))): + return "signature_invalid" + return None + + +# --------------------------------------------------------------------------- +# Canonical CBOR (identity spec §2.3) +# +# Values: int, bytes, str, list, Map (a list of (key, value) pairs), bool, +# None. Python's bool is an int subclass, so bools are tested first. +# --------------------------------------------------------------------------- + +class Map(list): + """A CBOR map as (key, value) pairs, in the order they were given.""" + + +class Refused(Exception): + def __init__(self, cls): + super().__init__(cls) + self.cls = cls + + +def _head(major, n): + if n < 24: + return bytes([major << 5 | n]) + for ai, width in ((24, 1), (25, 2), (26, 4), (27, 8)): + if n < 1 << (8 * width): + return bytes([major << 5 | ai]) + n.to_bytes(width, "big") + raise ValueError("argument does not fit in 64 bits") + + +def encode(v): + if v is None: + return b"\xf6" + if v is True: + return b"\xf5" + if v is False: + return b"\xf4" + if isinstance(v, int): + return _head(0, v) if v >= 0 else _head(1, -1 - v) + if isinstance(v, bytes): + return _head(2, len(v)) + v + if isinstance(v, str): + b = v.encode("utf-8") + return _head(3, len(b)) + b + if isinstance(v, Map): + pairs = sorted((encode(k), encode(x)) for k, x in v) + if any(pairs[i][0] == pairs[i + 1][0] for i in range(len(pairs) - 1)): + raise Refused("duplicate_map_key") + return _head(5, len(pairs)) + b"".join(k + x for k, x in pairs) + if isinstance(v, list): + return _head(4, len(v)) + b"".join(encode(x) for x in v) + raise TypeError(f"no canonical form for {type(v).__name__}") + + +def decode(data, strict=True): + """One canonical item, or Refused(class). strict=False accepts over-long + heads and unsorted maps; it exists only to show what a lenient + implementation would compute.""" + pos = 0 + + def need(n): + nonlocal pos + if n > len(data) - pos: + raise Refused("truncated") + out = data[pos:pos + n] + pos += n + return out + + def item(depth): + nonlocal pos + if depth > 128: + raise Refused("too_deep") + ib = need(1)[0] + major, ai = ib >> 5, ib & 31 + if major == 7: + if ai == 20: + return False + if ai == 21: + return True + if ai == 22: + return None + if ai in (25, 26, 27): + raise Refused("float_forbidden") + if ai in (28, 29, 30, 31): + raise Refused("malformed") + raise Refused("unsupported_simple_value") + if major == 6: + raise Refused("tag_forbidden") + if ai in (28, 29, 30): + raise Refused("malformed") + if ai == 31: + raise Refused("malformed" if major <= 1 else "indefinite_length") + if ai < 24: + n = ai + else: + width = {24: 1, 25: 2, 26: 4, 27: 8}[ai] + n = int.from_bytes(need(width), "big") + if strict and n < (24 if width == 1 else 1 << (8 * width // 2)): + raise Refused("non_shortest_integer" if major <= 1 else "non_shortest_length") + if major == 0: + return n + if major == 1: + return -1 - n + if major in (2, 3): + raw = need(n) + if major == 2: + return raw + try: + return raw.decode("utf-8") + except UnicodeDecodeError: + raise Refused("invalid_utf8") from None + if major == 4: + return [item(depth + 1) for _ in range(n)] + out, prev = Map(), None + for _ in range(n): + start = pos + k = item(depth + 1) + kb = data[start:pos] + if strict and prev is not None: + if kb == prev: + raise Refused("duplicate_map_key") + if kb < prev: + raise Refused("unsorted_map_keys") + prev = kb + out.append((k, item(depth + 1))) + return out + + v = item(0) + if pos != len(data): + raise Refused("trailing_bytes") + return v + + +def from_json(j): + """The pack's typed JSON: {"$bytes": hex} is a byte string.""" + if isinstance(j, dict): + if list(j) == ["$bytes"] and isinstance(j["$bytes"], str): + h = j["$bytes"] + if len(h) % 2 or any(c not in "0123456789abcdef" for c in h): + raise ValueError(f"$bytes must be lowercase hex: {h!r}") + return bytes.fromhex(h) + return Map((k, from_json(v)) for k, v in j.items()) + if isinstance(j, list): + return [from_json(x) for x in j] + if isinstance(j, float): + raise ValueError("floats have no canonical form") + return j + + +# --------------------------------------------------------------------------- +# SAIDs (identity spec §2.1, §2.2) +# --------------------------------------------------------------------------- + +PLACEHOLDER = "#" * 53 + + +def said_text(digest): + return "E" + base64.b32encode(digest).decode().rstrip("=").lower() + + +def parse_said(text): + if not (isinstance(text, str) and len(text) == 53 and text[0] == "E" + and all(c in "abcdefghijklmnopqrstuvwxyz234567" for c in text[1:])): + return None + digest = base64.b32decode(text[1:].upper() + "====") + return digest if said_text(digest) == text else None + + +def _with(obj, field, value): + return Map((k, value if k == field else v) for k, v in obj) + + +def derive(obj, field): + """(hashed bytes, digest, stamped object bytes).""" + if not isinstance(obj, Map): + raise Refused("not_a_map") + if not any(k == field for k, _ in obj): + raise Refused("missing_said_field") + hashed = encode(_with(obj, field, PLACEHOLDER)) + digest = blake3(hashed) + return hashed, digest, encode(_with(obj, field, said_text(digest))) + + +def verify_said(data, field): + """The digest the bytes state and hash to, or Refused(class).""" + obj = decode(data) + if not isinstance(obj, Map): + raise Refused("not_a_map") + stated = dict((k, v) for k, v in obj if isinstance(k, str)).get(field, Refused) + if stated is Refused: + raise Refused("missing_said_field") + digest = parse_said(stated) + if digest is None: + raise Refused("malformed_said") + if derive(obj, field)[1] != digest: + raise Refused("said_mismatch") + return digest + + +# --------------------------------------------------------------------------- +# The checks +# --------------------------------------------------------------------------- + +def refused_class(fn, *args): + try: + fn(*args) + except Refused as e: + return e.cls + return None + + +def check_canonicalisation(v): + if "input" in v: + b = encode(from_json(v["input"])) + assert b.hex() == v["expected_cbor_hex"], f"computed {b.hex()}" + assert encode(decode(b)) == b, "decode then encode changed the bytes" + else: + got = refused_class(decode, bytes.fromhex(v["input_cbor_hex"])) + assert got == v["error"], f"refused with {got}" + + +def check_said(v): + field = v["said_field"] + if "input" in v: + obj = from_json(v["input"]) + assert dict(obj).get(field) == PLACEHOLDER, "input must hold the placeholder" + hashed, digest, stamped = derive(obj, field) + assert hashed.hex() == v["canonical_cbor_hex"], "canonical_cbor_hex" + assert digest.hex() == v["blake3_hex"], "blake3_hex" + assert said_text(digest) == v["expected_said"], f"expected_said: computed {said_text(digest)}" + assert stamped.hex() == v["said_cbor_hex"], "said_cbor_hex" + assert verify_said(stamped, field) == digest, "said_cbor_hex does not verify" + else: + data = bytes.fromhex(v["input_cbor_hex"]) + got = refused_class(verify_said, data, field) + assert got == v["error"], f"refused with {got}" + lenient = decode(data, strict=False) + stated = dict(lenient)[field] + if "said_of_received_bytes" in v: + at = data.index(stated.encode()) + raw = data[:at] + PLACEHOLDER.encode() + data[at + 53:] + assert said_text(blake3(raw)) == v["said_of_received_bytes"], "said_of_received_bytes" + if "said_after_reencoding" in v: + assert said_text(derive(lenient, field)[1]) == v["said_after_reencoding"], "said_after_reencoding" + + +def check_signature(v, saids): + assert v["envelope_said"] in saids, "envelope_said is not a SAID vector's expected_said" + digest = parse_said(v["envelope_said"]) + assert digest is not None and digest.hex() == v["signed_message_hex"], "signed_message_hex" + seed = bytes.fromhex(v["signing_key_seed"]) + assert ed25519_public(seed).hex() == v["verifying_key"], "verifying_key" + sig = bytes.fromhex(v["signature"]) + got = ed25519_verify(bytes.fromhex(v["verifying_key"]), digest, sig) + if v["expected_verify"]: + assert got is None and v["error"] is None, f"a positive vector fails: {got}" + assert ed25519_sign(seed, digest) == sig, "the deterministic signature differs" + else: + assert got == v["error"], f"refused with {got}" + + +def check_lockstep(v): + n, frame = v["node_count"], v["frame_number"] + reports = v["node_hashes"] + assert 0 < len(reports) <= n + threshold = 2 * n // 3 + 1 + assert v["expected_threshold"] == threshold, "expected_threshold" + votes = {} + for h in reports.values(): + votes[h] = votes.get(h, 0) + 1 + winners = [h for h, c in votes.items() if c >= threshold] + winner = winners[0] if winners else None + assert v["expected_commit"] == (winner is not None), "expected_commit" + assert v["expected_state_hash"] == winner, "expected_state_hash" + dissenters = sorted(node for node, h in reports.items() if winner and h != winner) + outcomes = {} + for node, h in reports.items(): + if winner is None: + outcomes[node] = "cluster_split" + elif h != winner: + outcomes[node] = "minority_divergence" + else: + outcomes[node] = "commit_with_dissent" if dissenters else "clean_commit" + assert v["expected_outcomes"] == outcomes, f"expected_outcomes: computed {outcomes}" + entries = [] + if dissenters: + entries.append({"frame": frame, "dissenting_peers": dissenters, + "local_hash": winner, "supermajority_hash": winner}) + assert v["expected_dissent"] == entries, f"expected_dissent: computed {entries}" + + +def check_gossip(v): + held = [] + assert len(v["deliveries"]) == len(v["expected_results"]) + for d, want in zip(v["deliveries"], v["expected_results"]): + data = bytes.fromhex(d["cbor_hex"]) + try: + digest = verify_said(data, "d") + claimed = parse_said(d["claimed_said"]) + if claimed is None: + raise Refused("malformed_said") + if claimed != digest: + raise Refused("said_mismatch") + text = said_text(digest) + got = {"outcome": "duplicate" if text in held else "accepted", "said": text} + if text not in held: + held.append(text) + except Refused as e: + got = {"outcome": "rejected", "error": e.cls} + assert got == want, f"published {want}, computed {got}" + assert v["expected_held"] == held, f"expected_held: computed {held}" + + +def self_test(): + # BLAKE3 known answers: the empty input, and the official test-vector input + # pattern (byte i is i % 251) at lengths that cross block and chunk edges. + assert blake3(b"").hex() == "af1349b9f5f9a1a6a0404dea36dcc9499bcb25c9adc112b7cc9a93cae41f3262" + for n, want in BLAKE3_KAT: + got = blake3(bytes(i % 251 for i in range(n))).hex() + assert got == want, f"BLAKE3 known answer for length {n}: {got}" + # Ed25519 known answers: RFC 8032 §7.1 tests 1 and 2. + s1 = bytes.fromhex("9d61b19deffd5a60ba844af492ec2cc44449c5697b326919703bac031cae7f60") + assert ed25519_public(s1).hex() == "d75a980182b10ab7d54bfed3c964073a0ee172f3daa62325af021a68f707511a" + assert ed25519_sign(s1, b"").hex() == ( + "e5564300c360ac729086e2cc806e828a84877f1eb8e5d974d873e065224901555" + "fb8821590a33bacc61e39701cf9b46bd25bf5f0595bbe24655141438e7a100b") + s2 = bytes.fromhex("4ccd089b28ff96da9db6c346ec114e0f5b8a319f35aba624da8cf6ed4fb8a6fb") + sig2 = ed25519_sign(s2, b"\x72") + assert sig2.hex() == ( + "92a009a9f0d4cab8720e820b5f642540a2b27b5416503f8fb3762223ebdb69da" + "085ac1e43e15996e458f3613d0f11d8c387b2eaeb4302aeeb00d291612bb0c00") + assert ed25519_verify(ed25519_public(s2), b"\x72", sig2) is None + assert ed25519_verify(ed25519_public(s2), b"\x73", sig2) == "signature_invalid" + + +# BLAKE3 hashes of the official test-vector input pattern (byte i is i % 251), +# hash mode, 32 bytes, at lengths that cross block, chunk and tree edges. +# Computed with the BLAKE3 team's reference implementation (Rust crate +# `blake3` 1.8.7). +BLAKE3_KAT = [ + (1, "2d3adedff11b61f14c886e35afa036736dcd87a74d27b5c1510225d0f592e213"), + (63, "e9bc37a594daad83be9470df7f7b3798297c3d834ce80ba85d6e207627b7db7b"), + (64, "4eed7141ea4a5cd4b788606bd23f46e212af9cacebacdc7d1f4c6dc7f2511b98"), + (65, "de1e5fa0be70df6d2be8fffd0e99ceaa8eb6e8c93a63f2d8d1c30ecb6b263dee"), + (1023, "10108970eeda3eb932baac1428c7a2163b0e924c9a9e25b35bba72b28f70bd11"), + (1024, "42214739f095a406f3fc83deb889744ac00df831c10daa55189b5d121c855af7"), + (1025, "d00278ae47eb27b34faecf67b4fe263f82d5412916c1ffd97c8cb7fb814b8444"), + (2048, "e776b6028c7cd22a4d0ba182a8bf62205d2ef576467e838ed6f2529b85fba24a"), + (2049, "5f4d72f40d7a5f82b15ca2b2e44b1de3c2ef86c426c95c1af0b6879522563030"), + (3072, "b98cb0ff3623be03326b373de6b9095218513e64f1ee2edd2525c7ad1e5cffd2"), + (3073, "7124b49501012f81cc7f11ca069ec9226cecb8a2c850cfe644e327d22d3e1cd3"), + (4096, "015094013f57a5277b59d8475c0501042c0b642e531b0a1c8f58d2163229e969"), + (4097, "9b4052b38f1c5fc8b1f9ff7ac7b27cd242487b3d890d15c96a1c25b8aa0fb995"), + (5120, "9cadc15fed8b5d854562b26a9536d9707cadeda9b143978f319ab34230535833"), + (8193, "bab6c09cb8ce8cf459261398d2e7aef35700bf488116ceb94a36d0f5f1b7bc3b"), + (31744, "62b6960e1a44bcc1eb1a611a8d6235b6b4b78f32e7abc4fb4c6cdcce94895c47"), +] + +CHECKS = { + "canonicalisation_vectors.json": check_canonicalisation, + "said_vectors.json": check_said, + "lockstep_vectors.json": check_lockstep, + "gossip_vectors.json": check_gossip, +} + + +def main(pack): + self_test() + files = {f: json.loads((pack / f).read_text(encoding="utf-8")) + for f in list(CHECKS) + ["signature_vectors.json"]} + saids = {v["expected_said"] for v in files["said_vectors.json"] if "expected_said" in v} + failed = total = 0 + for name, vectors in files.items(): + names = [v["name"] for v in vectors] + if len(set(names)) != len(names): + print(f"FAIL {name}: duplicate vector names") + failed += 1 + for v in vectors: + total += 1 + try: + if name == "signature_vectors.json": + check_signature(v, saids) + else: + CHECKS[name](v) + print(f"ok {name:<32} {v['name']}") + except (AssertionError, Refused, KeyError, ValueError) as e: + failed += 1 + print(f"FAIL {name:<32} {v.get('name')}: {e!r}") + print(f"{total} vectors, {failed} failed") + return 1 if failed else 0 + + +if __name__ == "__main__": + default = pathlib.Path(__file__).resolve().parent.parent / "v1" + sys.exit(main(pathlib.Path(sys.argv[1]) if len(sys.argv) > 1 else default)) diff --git a/smart-byte/conformance/v1/canonicalisation_vectors.json b/smart-byte/conformance/v1/canonicalisation_vectors.json index bcd60616c..e68908f2e 100644 --- a/smart-byte/conformance/v1/canonicalisation_vectors.json +++ b/smart-byte/conformance/v1/canonicalisation_vectors.json @@ -1,17 +1,17 @@ [ { "name": "canon-001-sorted-map-keys", - "description": "Map keys must be sorted lexicographically in canonical CBOR. The input is given with keys out of order; the canonical encoding sorts them.", + "description": "Map keys are written in the bytewise order of their canonical encodings (identity spec §2.3). The input lists the keys out of order; the canonical encoding sorts them.", "input": { "z": 1, "a": 2, "m": 3 }, - "expected_cbor_hex": "a361610261186d036179016101" + "expected_cbor_hex": "a3616102616d03617a01" }, { "name": "canon-002-shortest-form-integers", - "description": "Integers must be encoded in their shortest form. No leading zeros, no over-large width selection. 0, 23, 24, 255, 256, 65535, 65536 cover the boundaries of CBOR's minor-info encoding.", + "description": "Integers use the shortest head that holds them: no leading zeros, no over-wide argument. 0, 23, 24, 255, 256, 65535 and 65536 sit on the boundaries of CBOR's argument widths.", "input": { "i0": 0, "i23": 23, @@ -21,34 +21,148 @@ "i65535": 65535, "i65536": 65536 }, - "expected_cbor_hex": "a76269300062693017636932341818636932351818ff63693235361901006935363533361903e8676935363533362ffffeeeebbbbcccc" + "expected_cbor_hex": "a7626930006369323317636932341818646932353518ff64693235361901006669363535333519ffff666936353533361a00010000" }, { "name": "canon-003-no-tags", - "description": "Canonical Smart Byte CBOR never uses CBOR tags. Application semantics carried in cargo fields are interpreted by layer-2 protocols; the substrate's wire format treats all values as untagged primitives. The input here is a plain byte string that an implementer might be tempted to wrap in tag 21 (expected base64url encoding) — the canonical form drops the tag.", + "description": "A byte string is written untagged, as major type 2. Wrapping bytes in tag 21 (\"expected base64url conversion\") is tempting, but canonical encoding forbids tags (identity spec §2.3). canon-015 is the tagged form, which a decoder refuses rather than silently stripping the tag.", "input": { - "payload": "AAECAwQFBgc=" + "payload": { + "$bytes": "0001020304050607" + } }, - "expected_cbor_hex": "a16770617964c6c6164480001020304050607" + "expected_cbor_hex": "a1677061796c6f6164480001020304050607" }, { - "name": "canon-004-deterministic-float-encoding", - "description": "Floating-point values are encoded in the shortest representation that preserves the value exactly. The spec recommends avoiding floats in cargo where possible (use fixed-point integers), but where they appear, the canonical encoding selects f16/f32/f64 by the round-trip test.", - "input": { - "exact_half": 1.5, - "needs_f32": 3.4028234663852886e38, - "needs_f64": 1.7976931348623157e308 - }, - "expected_cbor_hex": "a36a657861637421682dbcb1a96f3df9006a6e656564735f66333367fa7f7fffff6a6e656564735f6636341b7fefffffffffffff" + "name": "canon-004-floats-forbidden", + "description": "Floating-point values have no canonical form: identity spec §2.3 forbids them, and a value that would be a float elsewhere is carried as a fixed-point integer. The input is a map holding 1.5 as a half-precision float, and a decoder refuses it.", + "input_cbor_hex": "a16a65786163745f68616c66f93e00", + "error": "float_forbidden" }, { "name": "canon-005-bytes-not-text-for-byte-arrays", - "description": "Byte arrays MUST be encoded as CBOR major-type 2 (byte string), never as major-type 3 (text string). This matters for SAIDs, pubkeys, signatures, and any raw cryptographic material. The input shows three fields that look textual on the wire but are bytes by the spec: a SAID, a pubkey, a signature placeholder.", + "description": "Byte values are byte strings (major type 2) — never text, and never an array of small integers, which is what a generic serializer writes for a fixed-size byte array by default. The members hold a 32-byte digest, a 32-byte public key and a 64-byte signature. A SAID's text form (identity spec §2.2) is text; the digest it spells is bytes.", "input": { - "said": "0000000000000000000000000000000000000000000000000000000000000000", - "key": "1111111111111111111111111111111111111111111111111111111111111111", - "sig": "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222" + "said": { + "$bytes": "0000000000000000000000000000000000000000000000000000000000000000" + }, + "key": { + "$bytes": "1111111111111111111111111111111111111111111111111111111111111111" + }, + "sig": { + "$bytes": "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222" + } }, - "expected_cbor_hex": "a3637361696458200000000000000000000000000000000000000000000000000000000000000000636b657958201111111111111111111111111111111111111111111111111111111111111111637369675840222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222" + "expected_cbor_hex": "a3636b65795820111111111111111111111111111111111111111111111111111111111111111163736967584022222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222647361696458200000000000000000000000000000000000000000000000000000000000000000" + }, + { + "name": "canon-006-key-order-follows-encoded-bytes", + "description": "Keys sort by their encoded bytes, not their text (identity spec §2.3). As text \"aa\" comes before \"b\"; encoded, \"b\" (0x61 0x62) comes before \"aa\" (0x62 0x61 0x61), because a string's length is in its first byte. canon-012 is the text-ordered encoding, which a decoder refuses.", + "input": { + "aa": 1, + "b": 2 + }, + "expected_cbor_hex": "a261620262616101" + }, + { + "name": "canon-007-integer-boundaries-and-negatives", + "description": "Negative integers (major type 1) and the 32-bit/64-bit boundary. -1 is 0x20, and -24 is the last one-byte form; 4294967295 is the largest four-byte argument and 4294967296 the smallest eight-byte one.", + "input": [ + -1, + -24, + -25, + -256, + -257, + -65536, + -65537, + 4294967295, + 4294967296 + ], + "expected_cbor_hex": "892037381838ff39010039ffff3a000100001affffffff1b0000000100000000" + }, + { + "name": "canon-008-string-length-boundary", + "description": "A length uses the shortest head too. A 23-character text string carries its length in its first byte; a 24-character one needs the one-byte argument.", + "input": [ + "aaaaaaaaaaaaaaaaaaaaaaa", + "bbbbbbbbbbbbbbbbbbbbbbbb" + ], + "expected_cbor_hex": "827761616161616161616161616161616161616161616161617818626262626262626262626262626262626262626262626262" + }, + { + "name": "canon-009-empty-containers-and-simple-values", + "description": "An empty array, map, text string and byte string, and the simple values null, true and false — each a single byte.", + "input": { + "a": [], + "b": {}, + "c": "", + "d": { + "$bytes": "" + }, + "e": null, + "f": true, + "g": false + }, + "expected_cbor_hex": "a76161806162a06163606164406165f66166f56167f4" + }, + { + "name": "canon-010-non-shortest-integer-refused", + "description": "Zero written with a one-byte argument (0x18 0x00). Identity spec §2.3: \"a value of 0 is encoded as 0x00, never as 0x18 0x00\". A decoder refuses it rather than normalising it.", + "input_cbor_hex": "1800", + "error": "non_shortest_integer" + }, + { + "name": "canon-011-non-shortest-length-refused", + "description": "The one-character text \"a\" with its length in a one-byte argument (0x78 0x01) rather than in the head. A length has one canonical form too.", + "input_cbor_hex": "780161", + "error": "non_shortest_length" + }, + { + "name": "canon-012-text-ordered-keys-refused", + "description": "canon-006's map with its keys in text order (\"aa\" before \"b\"). The keys are distinct and the map is well-formed, but the order is not the canonical one, so the encoding is refused.", + "input_cbor_hex": "a262616101616202", + "error": "unsorted_map_keys" + }, + { + "name": "canon-013-duplicate-key-refused", + "description": "A map holding the key \"a\" twice. It has no canonical form, and which of the two values a lenient decoder keeps differs between implementations.", + "input_cbor_hex": "a2616101616102", + "error": "duplicate_map_key" + }, + { + "name": "canon-014-indefinite-length-refused", + "description": "An array written with an indefinite length (0x9f … 0xff). Identity spec §2.3 forbids indefinite-length items: the same array has a definite-length encoding.", + "input_cbor_hex": "9f0102ff", + "error": "indefinite_length" + }, + { + "name": "canon-015-tagged-byte-string-refused", + "description": "canon-003's map with its byte string wrapped in tag 21. Tags are forbidden (identity spec §2.3): the decoder refuses the input; it does not strip the tag and accept what is left.", + "input_cbor_hex": "a1677061796c6f6164d5480001020304050607", + "error": "tag_forbidden" + }, + { + "name": "canon-016-single-precision-float-refused", + "description": "A single-precision float (the largest finite one). Refused, as every float is.", + "input_cbor_hex": "fa7f7fffff", + "error": "float_forbidden" + }, + { + "name": "canon-017-double-precision-float-refused", + "description": "A double-precision float (the largest finite one). Refused, as every float is.", + "input_cbor_hex": "fb7fefffffffffffff", + "error": "float_forbidden" + }, + { + "name": "canon-018-trailing-bytes-refused", + "description": "Two items back to back. A canonical encoding is exactly one item: bytes after it would sit outside the value a SAID commits to, so the input is refused.", + "input_cbor_hex": "0101", + "error": "trailing_bytes" + }, + { + "name": "canon-019-invalid-utf8-refused", + "description": "A text string whose bytes (0xc3 0x28) are not UTF-8. CBOR text strings are UTF-8 (RFC 8949 §3.1), so the input is refused.", + "input_cbor_hex": "62c328", + "error": "invalid_utf8" } ] diff --git a/smart-byte/conformance/v1/gossip_vectors.json b/smart-byte/conformance/v1/gossip_vectors.json index d2d222b6b..5581f543a 100644 --- a/smart-byte/conformance/v1/gossip_vectors.json +++ b/smart-byte/conformance/v1/gossip_vectors.json @@ -1,65 +1,162 @@ [ { "name": "gossip-001-cross-node-propagation", - "description": "An envelope published at node A is observed at node B after a single round of gossip. The envelope's SAID is identical at both observation points; the gossip layer does not modify the wire bytes.", - "scenario": { - "publisher": "node_A", - "publish_frame": 184300, - "envelope_said": "EBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", - "envelope_canonical_cbor_hex": "a6677665727369", - "gossip_rounds": 1, - "observers": ["node_B"] - }, - "expected_observations": [ + "description": "An envelope published at node_A reaches node_B. node_B accepts it because its bytes are canonical and hash to the SAID they state, which is the SAID it was announced under, and it holds the bytes unchanged. Gossip carries the object itself, never a re-encoding of it.", + "receiver": "node_B", + "deliveries": [ { - "observer": "node_B", - "observed_said": "EBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", - "observation_frame": 184301, - "outcome": "accepted" + "from": "node_A", + "claimed_said": "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq", + "cbor_hex": "a5616478354574626c613563346e61787372657861666c787a72686d376672706b626463656769746b67686f6c663663636d74366977706f787165636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730c75657374696d617465645f6d6963726f6a6f756c65730f6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" } + ], + "expected_results": [ + { + "outcome": "accepted", + "said": "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq" + } + ], + "expected_held": [ + "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq" ] }, { "name": "gossip-002-said-conflict-rejected", - "description": "Two different envelopes are gossiped under the same SAID. The receiving node MUST reject the second envelope. The SAID is content-addressed (BLAKE3 of the canonical CBOR); two structurally-different envelopes producing the same SAID is either a collision (cryptographically negligible) or an adversarial substitution attempt. Either way, the receiver keeps the first observation and rejects the second.", - "scenario": { - "publisher_a": "node_A", - "publisher_b": "node_B", - "publish_frame": 184310, - "envelope_a_said": "ECCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC", - "envelope_a_canonical_cbor_hex": "a6677665727369aaaa", - "envelope_b_said": "ECCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC", - "envelope_b_canonical_cbor_hex": "a6677665727369bbbb", - "receiver": "node_C" - }, - "expected_outcome": { - "accepted": "envelope_a", - "rejected": "envelope_b", - "rejection_reason": "said_collision_envelope_a_already_observed", - "alert_class": "potential_substitution_attack" - } + "description": "node_C holds said-003's object when a second object arrives under the same SAID: said-003's with its reading changed and its SAID left as it was. Content addressing settles the conflict without a first-come rule: the newcomer does not hash to the SAID it states, so it is refused. Two different canonical objects cannot share a SAID short of a BLAKE3 collision.", + "receiver": "node_C", + "deliveries": [ + { + "from": "node_A", + "claimed_said": "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq", + "cbor_hex": "a56164783545676379356c6b77797034346b756d6175636e6f366f63736a7a763434747832636637646335327a3636376d757871376d7334687165636172676fa464747970657573656e736f722e74656d70657261747572652e76316973656e736f725f69646f726f6f66746f702d61727261792d376d63656c736975735f6d696c6c69195bb36d72656164696e675f6672616d651a0002cfac6a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730875657374696d617465645f6d6963726f6a6f756c65730a6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" + }, + { + "from": "node_B", + "claimed_said": "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq", + "cbor_hex": "a56164783545676379356c6b77797034346b756d6175636e6f366f63736a7a763434747832636637646335327a3636376d757871376d7334687165636172676fa464747970657573656e736f722e74656d70657261747572652e76316973656e736f725f69646f726f6f66746f702d61727261792d376d63656c736975735f6d696c6c691a0001869f6d72656164696e675f6672616d651a0002cfac6a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730875657374696d617465645f6d6963726f6a6f756c65730a6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" + } + ], + "expected_results": [ + { + "outcome": "accepted", + "said": "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq" + }, + { + "outcome": "rejected", + "error": "said_mismatch" + } + ], + "expected_held": [ + "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq" + ] }, { - "name": "gossip-003-sequence-gap-detection", - "description": "An envelope arrives at node B referencing a previous-history-hash that node B has not seen. Node B MUST detect the gap, request the missing history via gossip backfill, and not commit the envelope's transition until the gap is closed. The expected behaviour is *not* to reject the envelope — it is to defer acceptance pending backfill.", - "scenario": { - "publisher": "node_A", - "publish_frame": 184320, - "envelope_said": "EDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD", - "envelope_previous_history_hash": "EFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF", - "receiver": "node_B", - "receiver_known_history_heads": [ - "EAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", - "EBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" - ] - }, - "expected_outcome": { - "observer": "node_B", - "outcome": "deferred_pending_backfill", - "missing_history_hash": "EFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF", - "backfill_request_targets": ["node_A"], - "backfill_max_frames": 256, - "alert_class": "history_gap_detected" - } + "name": "gossip-003-backfill-answer-verified", + "description": "node_B lacks the object behind said-004's SAID and asks for it by identity: any party holding an identifier can request the rest from any peer (treatise §21.4). node_A answers with said-005's object, which is well-formed and self-consistent but not the one requested, so it is refused. node_D's answer hashes to the requested SAID and is accepted. Spec v1 leaves gossip propagation rules to a later revision (treatise §21.6); what it does fix is that an answer is checked against the identifier that was asked for.", + "receiver": "node_B", + "deliveries": [ + { + "from": "node_A", + "claimed_said": "Ees7ph2gwaouwjro6nb5wpwdoxhlxyeannprp7bprskconubd6a5a", + "cbor_hex": "a56164783545797a796c697769676d72707a63726834656379776863326976716d64377965376f6477777361376c796562696763627a61687a6165636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c6573182f75657374696d617465645f6d6963726f6a6f756c657318386a70726f76656e616e6365a463616c6701697369676e61747572655840222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222226d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111116e69737375616e63655f6672616d651904006f6f776e6572736869705f636861696e82a663616c6701656672616d6519041a697369676e617475726558404444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444469746f5f7075626b6579582033333333333333333333333333333333333333333333333333333333333333336b66726f6d5f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117570726576696f75735f686973746f72795f68617368582024bef3e8d603a964c5de687b67d86eb9d77c100d6be2ff85f19284e6d023f03aa663616c6701656672616d65190438697369676e617475726558406666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666669746f5f7075626b6579582055555555555555555555555555555555555555555555555555555555555555556b66726f6d5f7075626b6579582033333333333333333333333333333333333333333333333333333333333333337570726576696f75735f686973746f72795f68617368582048dd9261e8f0836e26d963e2924a0441c2ad1620654cbe76159581d10775aa15" + }, + { + "from": "node_D", + "claimed_said": "Ees7ph2gwaouwjro6nb5wpwdoxhlxyeannprp7bprskconubd6a5a", + "cbor_hex": "a561647835456573377068326777616f75776a726f366e6235777077646f78686c787965616e6e70727037627072736b636f6e7562643661356165636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730e75657374696d617465645f6d6963726f6a6f756c6573126a70726f76656e616e6365a463616c6701697369676e61747572655840222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222226d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111116e69737375616e63655f6672616d651904006f6f776e6572736869705f636861696e80" + } + ], + "expected_results": [ + { + "outcome": "rejected", + "error": "said_mismatch" + }, + { + "outcome": "accepted", + "said": "Ees7ph2gwaouwjro6nb5wpwdoxhlxyeannprp7bprskconubd6a5a" + } + ], + "expected_held": [ + "Ees7ph2gwaouwjro6nb5wpwdoxhlxyeannprp7bprskconubd6a5a" + ] + }, + { + "name": "gossip-004-duplicate-delivery-deduplicated", + "description": "The same object reaches node_B from two peers. The second delivery is a duplicate of a SAID already held and changes nothing: the overlay deduplicates by content address (treatise §11.4).", + "receiver": "node_B", + "deliveries": [ + { + "from": "node_A", + "claimed_said": "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq", + "cbor_hex": "a5616478354574626c613563346e61787372657861666c787a72686d376672706b626463656769746b67686f6c663663636d74366977706f787165636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730c75657374696d617465645f6d6963726f6a6f756c65730f6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" + }, + { + "from": "node_C", + "claimed_said": "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq", + "cbor_hex": "a5616478354574626c613563346e61787372657861666c787a72686d376672706b626463656769746b67686f6c663663636d74366977706f787165636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730c75657374696d617465645f6d6963726f6a6f756c65730f6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" + } + ], + "expected_results": [ + { + "outcome": "accepted", + "said": "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq" + }, + { + "outcome": "duplicate", + "said": "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq" + } + ], + "expected_held": [ + "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq" + ] + }, + { + "name": "gossip-005-non-canonical-copy-rejected", + "description": "said-009's bytes reach node_B announced under said-008's SAID, the SAID a re-encoding receiver would compute for them. The bytes are not canonical, so a conforming receiver refuses them (identity spec §2.5). Accepting them would let a second byte string circulate under said-008's SAID.", + "receiver": "node_B", + "deliveries": [ + { + "from": "node_A", + "claimed_said": "Ek2atam3xjn73papa7bkh5ltoypfci4umrl3k5hoxohi6ajmpnu4q", + "cbor_hex": "a261647835456b326174616d33786a6e37337061706137626b68356c746f797066636934756d726c336b35686f786f686936616a6d706e753471616e1801" + } + ], + "expected_results": [ + { + "outcome": "rejected", + "error": "non_shortest_integer" + } + ], + "expected_held": [] + }, + { + "name": "gossip-006-substitute-arriving-first-rejected", + "description": "gossip-002 in the other order: the altered object reaches node_C first and is refused, then said-003's object arrives and is accepted. Arrival order decides nothing; the content does.", + "receiver": "node_C", + "deliveries": [ + { + "from": "node_B", + "claimed_said": "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq", + "cbor_hex": "a56164783545676379356c6b77797034346b756d6175636e6f366f63736a7a763434747832636637646335327a3636376d757871376d7334687165636172676fa464747970657573656e736f722e74656d70657261747572652e76316973656e736f725f69646f726f6f66746f702d61727261792d376d63656c736975735f6d696c6c691a0001869f6d72656164696e675f6672616d651a0002cfac6a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730875657374696d617465645f6d6963726f6a6f756c65730a6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" + }, + { + "from": "node_A", + "claimed_said": "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq", + "cbor_hex": "a56164783545676379356c6b77797034346b756d6175636e6f366f63736a7a763434747832636637646335327a3636376d757871376d7334687165636172676fa464747970657573656e736f722e74656d70657261747572652e76316973656e736f725f69646f726f6f66746f702d61727261792d376d63656c736975735f6d696c6c69195bb36d72656164696e675f6672616d651a0002cfac6a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730875657374696d617465645f6d6963726f6a6f756c65730a6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" + } + ], + "expected_results": [ + { + "outcome": "rejected", + "error": "said_mismatch" + }, + { + "outcome": "accepted", + "said": "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq" + } + ], + "expected_held": [ + "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq" + ] } ] diff --git a/smart-byte/conformance/v1/lockstep_vectors.json b/smart-byte/conformance/v1/lockstep_vectors.json index 0ab5bfdf6..a5be91713 100644 --- a/smart-byte/conformance/v1/lockstep_vectors.json +++ b/smart-byte/conformance/v1/lockstep_vectors.json @@ -1,108 +1,134 @@ [ { "name": "lockstep-001-unanimous-commit", - "description": "All 4 nodes in the cluster agree on the post-frame state hash. The BFT threshold is floor(2n/3)+1 = 3, so 4-of-4 trivially commits. The expected_state_hash is the hash all four nodes reported.", + "description": "All four nodes report the same post-frame state hash. The threshold is floor(2·4/3)+1 = 3, so the frame commits, every node's outcome is a clean commit (treatise §13.3), and nothing is logged as dissent. State hashes in these vectors are opaque 32-byte values compared only for equality: §9.4 does not pin the state's serialization, so the commit rule is tested over the hashes nodes report.", "node_count": 4, "frame_number": 184237, - "ordered_transitions": [ - { - "byte_said": "EBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", - "from_pubkey": "1111111111111111111111111111111111111111111111111111111111111111", - "to_pubkey": "3333333333333333333333333333333333333333333333333333333333333333", - "signature": "44444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444" - }, - { - "byte_said": "ECCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC", - "from_pubkey": "3333333333333333333333333333333333333333333333333333333333333333", - "to_pubkey": "5555555555555555555555555555555555555555555555555555555555555555", - "signature": "66666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666" - } - ], "node_hashes": { - "node_1": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", - "node_2": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", - "node_3": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", - "node_4": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" + "node_1": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "node_2": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "node_3": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "node_4": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" }, + "expected_threshold": 3, "expected_commit": true, - "expected_state_hash": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "expected_state_hash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "expected_outcomes": { + "node_1": "clean_commit", + "node_2": "clean_commit", + "node_3": "clean_commit", + "node_4": "clean_commit" + }, "expected_dissent": [] }, { "name": "lockstep-002-supermajority-commit", - "description": "3 of 4 nodes agree on the post-frame state hash; the 4th node reports a different hash (either Byzantine or simply lagging on input ordering). floor(2*4/3)+1 = 3, threshold met, frame commits at the majority hash. The dissenting node's hash is recorded in the dissent log (spec §14) but does not block commit.", + "description": "Three of four nodes agree and node_4 reports another hash. The threshold of 3 is met and the frame commits at the majority hash. Nodes 1–3 commit with dissent and write one dissent-log entry naming node_4 (treatise §14.1). node_4's own outcome is minority divergence: it halts, and writes nothing.", "node_count": 4, "frame_number": 184238, - "ordered_transitions": [ - { - "byte_said": "EBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", - "from_pubkey": "5555555555555555555555555555555555555555555555555555555555555555", - "to_pubkey": "7777777777777777777777777777777777777777777777777777777777777777", - "signature": "88888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888" - } - ], "node_hashes": { - "node_1": "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", - "node_2": "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", - "node_3": "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", - "node_4": "DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD" + "node_1": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "node_2": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "node_3": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "node_4": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd" }, + "expected_threshold": 3, "expected_commit": true, - "expected_state_hash": "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", + "expected_state_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "expected_outcomes": { + "node_1": "commit_with_dissent", + "node_2": "commit_with_dissent", + "node_3": "commit_with_dissent", + "node_4": "minority_divergence" + }, "expected_dissent": [ { - "node": "node_4", - "reported_hash": "DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD", - "majority_hash": "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", - "frame": 184238 + "frame": 184238, + "dissenting_peers": [ + "node_4" + ], + "local_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "supermajority_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" } ] }, { "name": "lockstep-003-threshold-not-met", - "description": "Only 2 of 4 nodes agree on a single state hash. floor(2*4/3)+1 = 3, threshold not met, frame does NOT commit. The cluster must re-run input gathering or escalate per spec §13. The expected_state_hash is null because no single hash reached supermajority.", + "description": "No hash is reported by more than two of the four nodes. None reaches the threshold of 3, so the frame does not commit and every node's outcome is a cluster split (§13.3). No dissent is recorded: dissent is divergence from a committed hash (§14.2), and there is none.", "node_count": 4, "frame_number": 184239, - "ordered_transitions": [ - { - "byte_said": "ECCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC", - "from_pubkey": "7777777777777777777777777777777777777777777777777777777777777777", - "to_pubkey": "9999999999999999999999999999999999999999999999999999999999999999", - "signature": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" - } - ], "node_hashes": { - "node_1": "CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC", - "node_2": "CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC", - "node_3": "DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD", - "node_4": "EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE" + "node_1": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", + "node_2": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", + "node_3": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", + "node_4": "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee" }, + "expected_threshold": 3, "expected_commit": false, "expected_state_hash": null, + "expected_outcomes": { + "node_1": "cluster_split", + "node_2": "cluster_split", + "node_3": "cluster_split", + "node_4": "cluster_split" + }, + "expected_dissent": [] + }, + { + "name": "lockstep-004-three-nodes-one-liar-split", + "description": "The n = 3 exception (treatise §13.2). The threshold is 3, so one node reporting a different hash leaves 2 of the 3 votes the frame needs. The cluster detects the divergence but cannot tolerate it: no commit, and every node halts with a cluster split.", + "node_count": 3, + "frame_number": 184240, + "node_hashes": { + "node_1": "1111111111111111111111111111111111111111111111111111111111111111", + "node_2": "1111111111111111111111111111111111111111111111111111111111111111", + "node_3": "2222222222222222222222222222222222222222222222222222222222222222" + }, + "expected_threshold": 3, + "expected_commit": false, + "expected_state_hash": null, + "expected_outcomes": { + "node_1": "cluster_split", + "node_2": "cluster_split", + "node_3": "cluster_split" + }, + "expected_dissent": [] + }, + { + "name": "lockstep-005-seven-nodes-two-dissenters", + "description": "Seven nodes, threshold 5: five agree and two report hashes of their own. The frame commits. The dissent-log entry names both dissenters in ascending order, so every honest node writes the same entry (§14.2).", + "node_count": 7, + "frame_number": 184241, + "node_hashes": { + "node_1": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "node_2": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "node_3": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "node_4": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "node_5": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "node_6": "6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b", + "node_7": "7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c" + }, + "expected_threshold": 5, + "expected_commit": true, + "expected_state_hash": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "expected_outcomes": { + "node_1": "commit_with_dissent", + "node_2": "commit_with_dissent", + "node_3": "commit_with_dissent", + "node_4": "commit_with_dissent", + "node_5": "commit_with_dissent", + "node_6": "minority_divergence", + "node_7": "minority_divergence" + }, "expected_dissent": [ { - "node": "node_1", - "reported_hash": "CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC", - "majority_hash": null, - "frame": 184239 - }, - { - "node": "node_2", - "reported_hash": "CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC", - "majority_hash": null, - "frame": 184239 - }, - { - "node": "node_3", - "reported_hash": "DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD", - "majority_hash": null, - "frame": 184239 - }, - { - "node": "node_4", - "reported_hash": "EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE", - "majority_hash": null, - "frame": 184239 + "frame": 184241, + "dissenting_peers": [ + "node_6", + "node_7" + ], + "local_hash": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "supermajority_hash": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a" } ] } diff --git a/smart-byte/conformance/v1/regeneration.md b/smart-byte/conformance/v1/regeneration.md index b70a4169f..261093126 100644 --- a/smart-byte/conformance/v1/regeneration.md +++ b/smart-byte/conformance/v1/regeneration.md @@ -1,114 +1,204 @@ # Regenerating v1 Conformance Vectors -The vectors in this directory are **reference vectors generated alongside the -Rust reference implementation**, expected to be regenerated when -`smart-byte-rs` lands. Until that point, the `expected_*` fields in each -JSON file are placeholder values that are *internally consistent* (the same -envelope produces the same SAID across vectors) but are not yet -cryptographically computed. +Every value in this directory is computed by the Rust reference implementation +and checked by a second, independent implementation. Nothing in the pack is a +placeholder. -This file documents the regeneration procedure so the placeholders can be -replaced with real values in one mechanical pass when the reference -implementation is available. +- **Generator and verifier:** the `smart-byte-conformance` crate in + `../../smart-byte-rs`. Names, descriptions and inputs are written in its + `src/pack.rs`; every expected value is computed at generation time by + `smart-byte-core` (canonical CBOR, SAIDs, signatures) and + `smart-byte-lockstep` (the commit rule and dissent log). +- **Second implementation:** `../crosscheck/verify_v1.py`, Python standard + library only, written from the spec text in another language. It implements + BLAKE3 and Ed25519 itself and checks both against known answers before it + checks any vector. -## Prerequisites +## Procedure -- Rust toolchain pinned by `../../rust-toolchain.toml`. -- `smart-byte-rs` crate (private repo, to be open-sourced alongside spec - v1.0 freeze). -- `cargo run --bin gen-conformance-vectors` produces the v1 pack. - -## Regeneration procedure +From `smart-byte/smart-byte-rs`: ```bash -# From the smart-byte-rs checkout -cargo run --release --bin gen-conformance-vectors -- \ - --spec-version v1 \ - --output-dir /path/to/open-standards/smart-byte/conformance/v1 +# Write the pack. --output-dir defaults to this directory. +cargo run --release -p smart-byte-conformance --bin gen-conformance-vectors -- \ + --spec-version v1 --output-dir ../conformance/v1 + +# Check a pack on disk against the reference implementation, vector by vector. +cargo run --release -p smart-byte-conformance --bin verify-conformance-vectors -- \ + --pack ../conformance/v1 + +# Check it against the second implementation. +python3 ../conformance/crosscheck/verify_v1.py ../conformance/v1 ``` -The generator does the following, in order: +`gen-conformance-vectors --check` writes nothing and exits non-zero if any file +differs from the generator's output. -1. **Canonical CBOR vectors.** For each `input` JSON object, run the - canonical CBOR encoder and write the hex output to `expected_cbor_hex`. - The encoder uses RFC 8949 §4.2.1 (Core Deterministic Encoding) plus the - additional Smart Byte rules: byte-string for byte arrays, no tags, - sorted map keys. +To change a vector, edit `smart-byte-conformance/src/pack.rs` and regenerate. +Editing the JSON by hand fails CI: `cargo test` in `smart-byte-rs` requires the +published files to equal the generator's output byte for byte, verifies every +vector, and plants a defect in each file to prove the verifier would catch it. +The conformance gate runs the second implementation on every pack change. -2. **SAID vectors.** For each `envelope`, substitute a zero-byte placeholder - for the `said` field, canonical-CBOR-encode, BLAKE3-256 hash, and write - the result to `expected_said`. The SAID format is the KERI prefix-byte - scheme: `E` prefix byte (= BLAKE3-256) followed by the base64url - encoding of the 32-byte hash. The `canonical_cbor_hex` field is also - populated. +## What the generator computes -3. **Signature vectors.** For each positive vector, derive the Ed25519 - verifying key from the seed via the standard RFC 8032 procedure, sign - the `envelope_said` (32 bytes, raw, not the prefix-byte format) under - the signing key, and write the signature to the `signature` field. For - negative vectors, the generator does not modify the explicit - `signature` / `verifying_key` fields; it only verifies that the - `expected_verify: false` assertion holds when the named verifier runs - the vector. - -4. **Lockstep vectors.** For each vector, the generator runs the reference - BFT-commit logic on the provided `node_hashes` and confirms that - `expected_commit` and `expected_state_hash` match. For commits, it also - confirms the `expected_dissent` log matches what the reference - dissent-tracking logic produces. - -5. **Gossip vectors.** For each scenario, the generator simulates the - gossip propagation under spec §17 rules and confirms the - `expected_outcome` / `expected_observations` match the reference - gossip-engine output. +1. **Canonicalisation vectors.** A positive vector's `input` is encoded to + `expected_cbor_hex`, and the strict decoder must accept those bytes + unchanged. A negative vector's `input_cbor_hex` must be refused with its + `error` class; generation fails if the reference refuses it for any other + reason, so a negative vector cannot pass by accident. +2. **SAID vectors.** `input` holds the 53-`#` placeholder in its SAID field + `d`. Its canonical encoding is `canonical_cbor_hex` (the bytes that are + hashed), their BLAKE3-256 is `blake3_hex`, that digest as `E` + lowercase + base32 is `expected_said`, and the object with its SAID in place is + `said_cbor_hex`, which must verify. A negative vector's `input_cbor_hex` is + refused by SAID verification with its `error` class. +3. **Signature vectors.** The signed message is the 32 digest bytes a SAID + spells (`signed_message_hex`), not its 53-character text. A positive + vector's `signature` is the deterministic Ed25519 signature by + `signing_key_seed`; a negative vector is refused with its `error` class. +4. **Lockstep vectors.** The reported `node_hashes` are tallied against the + threshold floor(2n/3)+1 (treatise §13.1), giving each node its §13.3 outcome + and the §14.1 dissent-log entry the committing nodes write. +5. **Gossip vectors.** Each delivery goes through the intake rule the vectors + describe: bytes must be canonical, must hash to the SAID they state, and that + SAID must be the one they arrived under; a SAID already held is a duplicate. ## Invariant: cross-vector consistency -The generator MUST emit consistent values across vectors. In particular: +- Every `envelope_said` in `signature_vectors.json` is the `expected_said` of a + vector in `said_vectors.json` — the verifier and the second implementation + both enforce it. +- Gossip deliveries carry SAID vectors' `said_cbor_hex` bytes, except where a + vector alters them on purpose and says so. -- An envelope appearing in `said_vectors.json` and `lockstep_vectors.json` - has the same SAID in both files. -- A SAID appearing in `signature_vectors.json` as `envelope_said` and in - `said_vectors.json` as `expected_said` is identical. +## Cross-language check -The placeholder values in the current pack are internally consistent in -the same way; the regeneration replaces each placeholder family with the -real value computed from the canonical pipeline. - -## What is NOT regenerated - -- Vector `name` fields (stable across regenerations; implementations pin - to names, not array positions). -- Vector `description` fields (human prose, not generator output). -- The set of vectors (additions go through `VERSIONS.md` and the - CI gate). - -## Verifying a regeneration - -After regeneration, run: - -```bash -cargo run --release --bin verify-conformance-vectors -- \ - --pack /path/to/open-standards/smart-byte/conformance/v1 -``` - -The verifier reads each vector, re-runs the canonical pipeline, and -asserts byte-for-byte equality with the file contents. A clean exit -status confirms the pack is self-consistent and matches the reference -implementation's current behaviour. - -## Cross-language sanity check - -The vectors are the contract; the Rust reference implementation is the -*proximate* generator but not the canonical authority. Before merging a -regeneration, at least one additional implementation (currently planned: -Go and TypeScript) must independently produce the same `expected_*` -values from the same inputs. If implementations disagree, the disagreement -is resolved by appeal to the spec text, not by which implementation -emitted the value first. +This document has always required a second implementation to reproduce the +`expected_*` values before a regeneration merges. `../crosscheck/verify_v1.py` +is that implementation, and it reproduces every vector. Be clear about its +limits: it was written by the same author as the generator, from the spec text +rather than from the Rust. An implementation written by someone else is still +the stronger test. The Go and TypeScript implementations an earlier draft of +this document planned do not exist. ## Editorial change log +### 2026-09-30 — the pack regenerated from the reference implementation + +Every `expected_*` value was a placeholder. The SAIDs were runs of one repeated +character, every `canonical_cbor_hex` was the same 7-byte stub, and the +signatures were RFC 8032 test signatures over the empty message and over 0x72. +None of them signed any SAID. All five canonicalisation encodings were wrong, +and two of them were odd-length hex that could not be decoded at all. Every +value is now computed, and checked twice (see above). + +**The pack was regenerated in place, not by supersession.** The pack is DRAFT +(`../VERSIONS.md`), and in-place regeneration was the stated plan for its +placeholders. No vector replaced here could have been passed by any +implementation, so no implementation can have pinned one. The supersession rule +(`../README.md`, `VERSIONS.md` rule 4) governs v1 from its freeze onward. + +**Inputs that had to change before anything could be computed:** + +- JSON has no byte-string type, and the draft wrote byte values as hex text, + which is indistinguishable from text. Byte strings are now written + `{"$bytes": ""}` (see `../README.md`). +- SAID vectors: the SAID field is `d`, holding the 53-`#` placeholder (identity + spec §2.1, §2.2). The draft used a `said` member holding ``. + The object member is now `input` rather than `envelope`, and the `version` + member is gone: treatise §8.1 gives an envelope five fields, and the + algorithm identifier carries version information (`VERSIONS.md` rule 5). +- `said-005`'s `previous_history_hash` values were SAID-shaped filler. They are + now computed per treatise §10.1. The draft also gave said-005 the same SAID as + said-004. A §2 SAID covers the whole object, so they differ, and the + description now says why a byte's identity is fixed at its origin instead. +- `canon-003`'s byte string was given as base64 text; it is now bytes. + +**Vectors whose meaning contradicted the spec, corrected (and renamed where the +name stated the wrong rule):** + +- `canon-004-deterministic-float-encoding` → `canon-004-floats-forbidden`. The + draft encoded floats in their shortest exact width, but identity spec §2.3 + forbids floats. canon-004 is now a negative vector, and canon-016/017 cover + the other widths. +- `lockstep-003`. The draft logged dissent for all four nodes of a frame that + did not commit. Treatise §14.2 writes the dissent log only on a commit with + dissent, so there is no entry. Every lockstep dissent entry now has §14.1's + shape: one entry per frame, holding the dissenting peer set, the local hash + and the supermajority hash. Each vector also states its threshold and every + node's §13.3 outcome. +- Lockstep vectors no longer carry `ordered_transitions`. They were never + inputs to the commit rule, which is decided over the hashes nodes report + (§9.4 does not pin the state's serialization), and their "signatures" were + filler. +- `gossip-002-said-conflict-rejected` keeps its name but not its rule. The + draft settled a "collision" by keeping whichever object arrived first. Two + different canonical objects cannot share a SAID short of a BLAKE3 collision, + so the newcomer is refused because it does not hash to the SAID it states, + whatever the arrival order (gossip-006 is the reverse order). +- `gossip-003-sequence-gap-detection` → `gossip-003-backfill-answer-verified`. + The draft asserted a 256-frame backfill limit, request targets and alert + classes that no spec text defines; treatise §21.6 defers gossip rules to a + later revision. The replacement tests what the spec does fix: an answer to a + request by identity is checked against that identity (§21.4, §10.2 step 1). +- `gossip-001` no longer asserts publish/observation frames or round counts: + gossip is not frame-synchronous in v1. +- Signature vectors: sig-003 and sig-004 now share the error class + `signature_invalid`. A verifier cannot tell "wrong SAID" from "wrong key"; + both are the same failed equation. sig-004's key is now derived from its + seed; the draft's key was derived from nothing. sig-005's class is + `malformed_signature`, and its timing claim is gone: a vector cannot test + timing, and this one requires the structural failure to be reported + distinctly. + +**Vectors added:** canon-006 to canon-019 (each §2.3 rule, accepted and +refused); said-006 to said-011 (the suite identity spec §2.5 names: the +empty-map case, the one-field case, the worked example, two adversarial +non-canonical encodings, and a changed-content mismatch); sig-006 (a +non-canonical S, RFC 8032 §5.1.7); lockstep-004 (the n = 3 exception, §13.2); +lockstep-005 (two dissenters); gossip-004 to gossip-006. + +**Spec text changed with it:** + +- Identity spec §2.4 printed an invented digest and SAID, and counted the + encoding as 62 bytes; it is 61. It now prints said-008's values. +- §2.5 names the vectors. +- §6 pointed at modules that did not exist. It now says where each part lives, + including that the key-event log does not yet derive §2 SAIDs. + +**Reference implementation changed with it:** + +- `smart-byte-core` gained canonical CBOR with a strict decoder, the §2 SAID + text form with derivation, verification and per-key SAIDs, and SAID signature + verification. +- `smart-byte-lockstep` gained the §13/§14 tally. `Cluster::step` uses it and + now keeps a dissent log. +- `smart-byte-libp2p` now checks received envelopes against the ids they state + on every gossip receive path, and checks a fetch's answer against the SAID + that was requested. Before, neither path checked, so a peer could answer a + fetch for one envelope with another. + +**Found and not fixed here.** These are normative or larger decisions: + +1. The reference `Envelope` type and the key-event log in + `smart-byte-keri-witness` derive identifiers with a general-purpose CBOR + serializer and a zeroed placeholder, not §2, so their ids are not §2 SAIDs. + Migrating them changes every identifier they produce. For `Envelope` it also + means first settling its wire schema against treatise §8: its wall-clock + issuance time and string-tagged cargo diverge from §8.3, §8.5 and §8.7. +2. Identity spec §4 step 1 replaces only `d` with the placeholder, but an + inception event's `i` is derived from its own `d` (§3.1), so the inception + SAID cannot be computed as written. The reference key-event log blanks `i` + too. The spec should say what to do. +3. §3.4 writes the per-key object's `k` as `""`, while §3.3 says a key is + prefixed with a one-byte algorithm identifier. said-007 carries `k` as a + byte string. If text was intended, said-007 changes. +4. The state serialization behind a post-frame hash (§9.4 step 6) and the + transition payload behind a history hash (§10.1) are not pinned, so no + vector can pin a real state hash or history hash. said-005 illustrates the + chain step with the payload taken as the transition's canonical encoding. + ### 2026-09-21 — spec reference corrected; no wire-format change `spec/identity_and_key_rotation.md` pointed three times at diff --git a/smart-byte/conformance/v1/said_vectors.json b/smart-byte/conformance/v1/said_vectors.json index 6722cfd6f..564ced49f 100644 --- a/smart-byte/conformance/v1/said_vectors.json +++ b/smart-byte/conformance/v1/said_vectors.json @@ -1,10 +1,10 @@ [ { "name": "said-001-empty-envelope", - "description": "SAID computation for the minimal envelope: schema-required fields only, no cargo, no provenance, no ownership transitions. This is the degenerate case used to verify that an implementation handles the canonical-CBOR + BLAKE3-256 pipeline on a payload with no application data.", - "envelope": { - "version": 1, - "said": "", + "description": "The smallest envelope-shaped object (treatise §8.1): no provenance, no cargo, no ownership transitions, zero joule cost. It exercises the whole pipeline — placeholder, canonical CBOR, BLAKE3-256, `E` + base32 — on an object with no application data.", + "said_field": "d", + "input": { + "d": "#####################################################", "provenance": null, "ownership_chain": [], "cargo": null, @@ -13,36 +13,42 @@ "estimated_microjoules": 0 } }, - "canonical_cbor_hex": "a6677665727369", - "expected_said": "EAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" + "canonical_cbor_hex": "a561647835232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232365636172676ff66a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730075657374696d617465645f6d6963726f6a6f756c6573006a70726f76656e616e6365f66f6f776e6572736869705f636861696e80", + "blake3_hex": "f715776741e2a8a4157d584ed68c4574b4185c693bfb635ff93efa48be55d736", + "expected_said": "E64kxoz2b4kukifl5lbhnndcfos2bqxdjhp5wgx7zh35erpsv243a", + "said_cbor_hex": "a5616478354536346b786f7a3262346b756b69666c356c62686e6e6463666f7332627178646a687035776778377a68333565727073763234336165636172676ff66a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730075657374696d617465645f6d6963726f6a6f756c6573006a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" }, { "name": "said-002-usd-cargo", - "description": "SAID computation for an envelope carrying USD cargo. Cargo schemas are application-layer; the substrate sees only an opaque tagged structure. This vector fixes the cargo to a 100.00 USD denomination issued by issuer pubkey 0x11...11.", - "envelope": { - "version": 1, - "said": "", + "description": "An envelope-shaped object carrying a USD cargo of 100.00 (100000000 micros) from issuer key 0x11…11. Cargo is opaque to the substrate; here it is a map whose members include a byte-string key.", + "said_field": "d", + "input": { + "d": "#####################################################", "provenance": null, "ownership_chain": [], "cargo": { "type": "usd.cargo.v1", "denomination_micros": 100000000, - "issuer_pubkey": "1111111111111111111111111111111111111111111111111111111111111111" + "issuer_pubkey": { + "$bytes": "1111111111111111111111111111111111111111111111111111111111111111" + } }, "joule_cost": { "measured_microjoules": 12, "estimated_microjoules": 15 } }, - "canonical_cbor_hex": "a6677665727369", - "expected_said": "EBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" + "canonical_cbor_hex": "a561647835232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232365636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730c75657374696d617465645f6d6963726f6a6f756c65730f6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80", + "blake3_hex": "98560e8b8d05e5125c055df313b3e58bd411888644d463b965f084c9f9167baf", + "expected_said": "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq", + "said_cbor_hex": "a5616478354574626c613563346e61787372657861666c787a72686d376672706b626463656769746b67686f6c663663636d74366977706f787165636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730c75657374696d617465645f6d6963726f6a6f756c65730f6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" }, { "name": "said-003-custom-cargo", - "description": "SAID computation for an envelope carrying a non-money cargo type (in this case a temperature sensor reading). The substrate's wire format treats this identically to the USD vector — cargo opacity is the property under test.", - "envelope": { - "version": 1, - "said": "", + "description": "The same shape carrying a non-money cargo: a temperature reading. The pipeline treats it exactly as it treats said-002 — cargo opacity is the property under test.", + "said_field": "d", + "input": { + "d": "#####################################################", "provenance": null, "ownership_chain": [], "cargo": { @@ -56,76 +62,175 @@ "estimated_microjoules": 10 } }, - "canonical_cbor_hex": "a6677665727369", - "expected_said": "ECCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC" + "canonical_cbor_hex": "a561647835232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232365636172676fa464747970657573656e736f722e74656d70657261747572652e76316973656e736f725f69646f726f6f66746f702d61727261792d376d63656c736975735f6d696c6c69195bb36d72656164696e675f6672616d651a0002cfac6a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730875657374696d617465645f6d6963726f6a6f756c65730a6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80", + "blake3_hex": "30b1d5aad87f38aa3014135de70a49cd79c9df422fc62eeb3ef7d94bc3ec970f", + "expected_said": "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq", + "said_cbor_hex": "a56164783545676379356c6b77797034346b756d6175636e6f366f63736a7a763434747832636637646335327a3636376d757871376d7334687165636172676fa464747970657573656e736f722e74656d70657261747572652e76316973656e736f725f69646f726f6f66746f702d61727261792d376d63656c736975735f6d696c6c69195bb36d72656164696e675f6672616d651a0002cfac6a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730875657374696d617465645f6d6963726f6a6f756c65730a6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" }, { "name": "said-004-with-provenance", - "description": "SAID computation for an envelope with a provenance attestation. Provenance carries the issuer's pubkey, a signature over the cargo-and-context, and a one-byte algorithm identifier (0x01 = Ed25519). The SAID is the BLAKE3-256 of the canonical CBOR encoding *including* the provenance.", - "envelope": { - "version": 1, - "said": "", + "description": "An envelope-shaped object with a provenance attestation: algorithm identifier 0x01 (Ed25519), issuer key, issuance frame and signature. The SAID covers the provenance. The key and signature are fixed filler bytes: this vector tests SAID derivation, not signature validity, which signature_vectors.json covers.", + "said_field": "d", + "input": { + "d": "#####################################################", "provenance": { "alg": 1, - "issuer_pubkey": "1111111111111111111111111111111111111111111111111111111111111111", + "issuer_pubkey": { + "$bytes": "1111111111111111111111111111111111111111111111111111111111111111" + }, "issuance_frame": 1024, - "signature": "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222" + "signature": { + "$bytes": "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222" + } }, "ownership_chain": [], "cargo": { "type": "usd.cargo.v1", "denomination_micros": 100000000, - "issuer_pubkey": "1111111111111111111111111111111111111111111111111111111111111111" + "issuer_pubkey": { + "$bytes": "1111111111111111111111111111111111111111111111111111111111111111" + } }, "joule_cost": { "measured_microjoules": 14, "estimated_microjoules": 18 } }, - "canonical_cbor_hex": "a6677665727369", - "expected_said": "EDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD" + "canonical_cbor_hex": "a561647835232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232365636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730e75657374696d617465645f6d6963726f6a6f756c6573126a70726f76656e616e6365a463616c6701697369676e61747572655840222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222226d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111116e69737375616e63655f6672616d651904006f6f776e6572736869705f636861696e80", + "blake3_hex": "24bef3e8d603a964c5de687b67d86eb9d77c100d6be2ff85f19284e6d023f03a", + "expected_said": "Ees7ph2gwaouwjro6nb5wpwdoxhlxyeannprp7bprskconubd6a5a", + "said_cbor_hex": "a561647835456573377068326777616f75776a726f366e6235777077646f78686c787965616e6e70727037627072736b636f6e7562643661356165636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730e75657374696d617465645f6d6963726f6a6f756c6573126a70726f76656e616e6365a463616c6701697369676e61747572655840222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222226d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111116e69737375616e63655f6672616d651904006f6f776e6572736869705f636861696e80" }, { "name": "said-005-with-ownership-transitions", - "description": "SAID computation for an envelope with two completed ownership transitions. Each transition signs (byte SAID, previous history hash, from pubkey, to pubkey, frame number). Note that the byte's SAID is computed over the *origin* envelope and never changes; this vector represents the envelope *as it would be encoded in the wire format when fetched at frame 1100*, with two transitions in its history.", - "envelope": { - "version": 1, - "said": "", + "description": "said-004's byte after two transfers, as fetched at frame 1100. The first transition's previous_history_hash is the digest of said-004's SAID, because a byte's identity is its initial history hash (treatise §10.1); the second is blake3 of that digest followed by the first transition's canonical encoding, illustrating §10.1's chain step (spec v1 does not pin the payload's serialization). This object's SAID differs from said-004's: a SAID is a function of the whole object, which is why a byte's identity is fixed at its origin (treatise §8.2) rather than recomputed as transitions accrue.", + "said_field": "d", + "input": { + "d": "#####################################################", "provenance": { "alg": 1, - "issuer_pubkey": "1111111111111111111111111111111111111111111111111111111111111111", + "issuer_pubkey": { + "$bytes": "1111111111111111111111111111111111111111111111111111111111111111" + }, "issuance_frame": 1024, - "signature": "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222" + "signature": { + "$bytes": "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222" + } }, "ownership_chain": [ { "alg": 1, - "from_pubkey": "1111111111111111111111111111111111111111111111111111111111111111", - "to_pubkey": "3333333333333333333333333333333333333333333333333333333333333333", + "from_pubkey": { + "$bytes": "1111111111111111111111111111111111111111111111111111111111111111" + }, + "to_pubkey": { + "$bytes": "3333333333333333333333333333333333333333333333333333333333333333" + }, "frame": 1050, - "previous_history_hash": "EDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD", - "signature": "44444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444" + "previous_history_hash": { + "$bytes": "24bef3e8d603a964c5de687b67d86eb9d77c100d6be2ff85f19284e6d023f03a" + }, + "signature": { + "$bytes": "44444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444" + } }, { "alg": 1, - "from_pubkey": "3333333333333333333333333333333333333333333333333333333333333333", - "to_pubkey": "5555555555555555555555555555555555555555555555555555555555555555", + "from_pubkey": { + "$bytes": "3333333333333333333333333333333333333333333333333333333333333333" + }, + "to_pubkey": { + "$bytes": "5555555555555555555555555555555555555555555555555555555555555555" + }, "frame": 1080, - "previous_history_hash": "EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE", - "signature": "66666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666" + "previous_history_hash": { + "$bytes": "48dd9261e8f0836e26d963e2924a0441c2ad1620654cbe76159581d10775aa15" + }, + "signature": { + "$bytes": "66666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666" + } } ], "cargo": { "type": "usd.cargo.v1", "denomination_micros": 100000000, - "issuer_pubkey": "1111111111111111111111111111111111111111111111111111111111111111" + "issuer_pubkey": { + "$bytes": "1111111111111111111111111111111111111111111111111111111111111111" + } }, "joule_cost": { "measured_microjoules": 47, "estimated_microjoules": 56 } }, - "canonical_cbor_hex": "a6677665727369", - "expected_said": "EDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD" + "canonical_cbor_hex": "a561647835232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232365636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c6573182f75657374696d617465645f6d6963726f6a6f756c657318386a70726f76656e616e6365a463616c6701697369676e61747572655840222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222226d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111116e69737375616e63655f6672616d651904006f6f776e6572736869705f636861696e82a663616c6701656672616d6519041a697369676e617475726558404444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444469746f5f7075626b6579582033333333333333333333333333333333333333333333333333333333333333336b66726f6d5f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117570726576696f75735f686973746f72795f68617368582024bef3e8d603a964c5de687b67d86eb9d77c100d6be2ff85f19284e6d023f03aa663616c6701656672616d65190438697369676e617475726558406666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666669746f5f7075626b6579582055555555555555555555555555555555555555555555555555555555555555556b66726f6d5f7075626b6579582033333333333333333333333333333333333333333333333333333333333333337570726576696f75735f686973746f72795f68617368582048dd9261e8f0836e26d963e2924a0441c2ad1620654cbe76159581d10775aa15", + "blake3_hex": "c670b45906645f9144fc20b1638b48ac183fe09f70ed6903ebc10283083901f2", + "expected_said": "Eyzyliwigmrpzcrh4ecywhc2ivqmd7ye7odwwsa7lyebigcbzahza", + "said_cbor_hex": "a56164783545797a796c697769676d72707a63726834656379776863326976716d64377965376f6477777361376c796562696763627a61687a6165636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c6573182f75657374696d617465645f6d6963726f6a6f756c657318386a70726f76656e616e6365a463616c6701697369676e61747572655840222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222226d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111116e69737375616e63655f6672616d651904006f6f776e6572736869705f636861696e82a663616c6701656672616d6519041a697369676e617475726558404444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444469746f5f7075626b6579582033333333333333333333333333333333333333333333333333333333333333336b66726f6d5f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117570726576696f75735f686973746f72795f68617368582024bef3e8d603a964c5de687b67d86eb9d77c100d6be2ff85f19284e6d023f03aa663616c6701656672616d65190438697369676e617475726558406666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666669746f5f7075626b6579582055555555555555555555555555555555555555555555555555555555555555556b66726f6d5f7075626b6579582033333333333333333333333333333333333333333333333333333333333333337570726576696f75735f686973746f72795f68617368582048dd9261e8f0836e26d963e2924a0441c2ad1620654cbe76159581d10775aa15" + }, + { + "name": "said-006-only-said-field", + "description": "The empty-map case of identity spec §2.5: an object whose only member is its SAID field. It is the smallest object a SAID can be derived over — a truly empty map has nowhere to put one.", + "said_field": "d", + "input": { + "d": "#####################################################" + }, + "canonical_cbor_hex": "a1616478352323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323", + "blake3_hex": "e0e9858d678259ae81196c47a1de59f52e53dfd5c413a44dd336ce901eb00291", + "expected_said": "E4duyldlhqjm25aiznrd2dxsz6uxfhx6vyqj2itotg3hjahvqakiq", + "said_cbor_hex": "a16164783545346475796c646c68716a6d323561697a6e7264326478737a367578666878367679716a3269746f746733686a61687671616b6971" + }, + { + "name": "said-007-one-field-per-key", + "description": "The one-field case: identity spec §3.4's per-key object {d, k}, whose SAID is a next-key commitment. k is the algorithm identifier 0x01 (Ed25519) followed by the public key of RFC 8032's first test seed, carried as a byte string.", + "said_field": "d", + "input": { + "d": "#####################################################", + "k": { + "$bytes": "01d75a980182b10ab7d54bfed3c964073a0ee172f3daa62325af021a68f707511a" + } + }, + "canonical_cbor_hex": "a2616478352323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323616b582101d75a980182b10ab7d54bfed3c964073a0ee172f3daa62325af021a68f707511a", + "blake3_hex": "c9357de555002f5bd3c70ffaa40d0d8a99316614f6c3e8372136cdfe93c76da2", + "expected_said": "Eze2x3zkvaaxvxu6hb75kidinrkmtczqu63b6qnzbg3g75e6hnwra", + "said_cbor_hex": "a261647835457a653278337a6b7661617876787536686237356b6964696e726b6d74637a717536336236716e7a6267336737356536686e777261616b582101d75a980182b10ab7d54bfed3c964073a0ee172f3daa62325af021a68f707511a" + }, + { + "name": "said-008-spec-worked-example", + "description": "The worked example of identity spec §2.4: {\"d\": placeholder, \"n\": 1}. Its canonical encoding is 61 bytes, and §2.4 prints the digest and SAID given here.", + "said_field": "d", + "input": { + "d": "#####################################################", + "n": 1 + }, + "canonical_cbor_hex": "a2616478352323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323616e01", + "blake3_hex": "56813033774b7fb781e0f8547eae6ec3ca24728c8af6ae9dd771d1e0258f6d39", + "expected_said": "Ek2atam3xjn73papa7bkh5ltoypfci4umrl3k5hoxohi6ajmpnu4q", + "said_cbor_hex": "a261647835456b326174616d33786a6e37337061706137626b68356c746f797066636934756d726c336b35686f786f686936616a6d706e753471616e01" + }, + { + "name": "said-009-non-canonical-copy-rejected", + "description": "The adversarial case of identity spec §2.5, aimed at re-encoding. These are said-008's bytes with n written 0x18 0x01 instead of 0x01, still stating said-008's SAID. A verifier that re-encodes before hashing gets said-008's bytes back and accepts, so a second byte string would verify under said-008's SAID. A conforming verifier refuses the input as non-canonical before computing anything.", + "said_field": "d", + "input_cbor_hex": "a261647835456b326174616d33786a6e37337061706137626b68356c746f797066636934756d726c336b35686f786f686936616a6d706e753471616e1801", + "error": "non_shortest_integer", + "said_of_received_bytes": "Eiq35nwixqgv23jmqme6sgpiismnq7vv42z33qodyypexl3muy4ma", + "said_after_reencoding": "Ek2atam3xjn73papa7bkh5ltoypfci4umrl3k5hoxohi6ajmpnu4q" + }, + { + "name": "said-010-self-consistent-non-canonical-rejected", + "description": "The adversarial case aimed the other way. said-008's object with its keys in reverse order, stating the SAID of exactly these bytes (placeholder substituted in place). A verifier that hashes what it received, without checking canonical form, accepts it. A conforming verifier refuses it as non-canonical.", + "said_field": "d", + "input_cbor_hex": "a2616e016164783545686b706b787576766a346266617272357677366e70716f3665746e347a79746d797972373462666534646c6e6f64776a68646971", + "error": "unsorted_map_keys", + "said_of_received_bytes": "Ehkpkxuvvj4bfarr5vw6npqo6etn4zytmyyr74bfe4dlnodwjhdiq", + "said_after_reencoding": "Ek2atam3xjn73papa7bkh5ltoypfci4umrl3k5hoxohi6ajmpnu4q" + }, + { + "name": "said-011-changed-content-rejected", + "description": "said-008's object with n changed from 1 to 2 and its SAID left as it was. The bytes are canonical and well-formed, but they state a SAID they do not hash to.", + "said_field": "d", + "input_cbor_hex": "a261647835456b326174616d33786a6e37337061706137626b68356c746f797066636934756d726c336b35686f786f686936616a6d706e753471616e02", + "error": "said_mismatch" } ] diff --git a/smart-byte/conformance/v1/signature_vectors.json b/smart-byte/conformance/v1/signature_vectors.json index 11d1114c3..6437dabe3 100644 --- a/smart-byte/conformance/v1/signature_vectors.json +++ b/smart-byte/conformance/v1/signature_vectors.json @@ -1,52 +1,68 @@ [ { "name": "sig-001-known-key-known-said", - "description": "Positive: sign a known envelope SAID with a known Ed25519 signing key, expect a deterministic signature. Ed25519 (RFC 8032) is deterministic, so the signature is reproducible across implementations.", - "envelope_said": "EBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", + "description": "Sign said-002's SAID with the key of RFC 8032's first test seed. The message is the 32 digest bytes the SAID spells (signed_message_hex), not its 53-character text. Ed25519 is deterministic, so every implementation produces exactly this signature.", + "envelope_said": "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq", + "signed_message_hex": "98560e8b8d05e5125c055df313b3e58bd411888644d463b965f084c9f9167baf", "signing_key_seed": "9d61b19deffd5a60ba844af492ec2cc44449c5697b326919703bac031cae7f60", - "verifying_key": "d75a980182b10ab7d54bfed3c964073a0ee172f3daa62325af021a68f707511a", - "signature": "e5564300c360ac729086e2cc806e828a84877f1eb8e5d974d873e065224901555fb8821590a33bacc61e39701cf9b46bd25bf5f0595bbe24655141438e7a100b", + "verifying_key": "d75a980182b10ab7d54bfed3c964073a0ee172f3daa62325af021a68f707511a", + "signature": "ceadf887b7391770a3c4a4a4cf2062f2e41db1447851cf2a843bf7b7718ae7ec8a534846a1bf46fbdedf3f9ee4a5832c9406cf33fd598fe6d520f98946e09b04", "expected_verify": true, "error": null }, { "name": "sig-002-verify-positive", - "description": "Positive: verify a signature against the envelope SAID and verifying key it was produced for. Result MUST be true.", - "envelope_said": "EAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "description": "A signature over said-001's SAID by the key of RFC 8032's second test seed. It verifies under that key.", + "envelope_said": "E64kxoz2b4kukifl5lbhnndcfos2bqxdjhp5wgx7zh35erpsv243a", + "signed_message_hex": "f715776741e2a8a4157d584ed68c4574b4185c693bfb635ff93efa48be55d736", "signing_key_seed": "4ccd089b28ff96da9db6c346ec114e0f5b8a319f35aba624da8cf6ed4fb8a6fb", - "verifying_key": "3d4017c3e843895a92b70aa74d1b7ebc9c982ccf2ec4968cc0cd55f12af4660c", - "signature": "92a009a9f0d4cab8720e820b5f642540a2b27b5416503f8fb3762223ebdb69da085ac1e43e15996e458f3613d0f11d8c387b2eaeb4302aeeb00d291612bb0c00", + "verifying_key": "3d4017c3e843895a92b70aa74d1b7ebc9c982ccf2ec4968cc0cd55f12af4660c", + "signature": "ce9b9e7185967276bebe6a9c12f3570d821bcb7184e9c703bd278c1fa44ece65c0fb5b0855fe3fca5eb836d3f41b568a230af3156da0440cfaf5d4acd7123506", "expected_verify": true, "error": null }, { "name": "sig-003-tampered-envelope", - "description": "Negative: verify a signature against a *different* envelope SAID than the one it was produced for. The signature is structurally valid but binds a different SAID; verification MUST return false. This catches the most common attack: an adversary swaps the envelope and forwards the original signature.", - "envelope_said": "EFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF", + "description": "sig-002's signature presented with said-003's SAID: an adversary swaps the envelope and forwards the original signature. The signature is well-formed but binds another SAID, so verification MUST fail.", + "envelope_said": "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq", + "signed_message_hex": "30b1d5aad87f38aa3014135de70a49cd79c9df422fc62eeb3ef7d94bc3ec970f", "signing_key_seed": "4ccd089b28ff96da9db6c346ec114e0f5b8a319f35aba624da8cf6ed4fb8a6fb", - "verifying_key": "3d4017c3e843895a92b70aa74d1b7ebc9c982ccf2ec4968cc0cd55f12af4660c", - "signature": "92a009a9f0d4cab8720e820b5f642540a2b27b5416503f8fb3762223ebdb69da085ac1e43e15996e458f3613d0f11d8c387b2eaeb4302aeeb00d291612bb0c00", + "verifying_key": "3d4017c3e843895a92b70aa74d1b7ebc9c982ccf2ec4968cc0cd55f12af4660c", + "signature": "ce9b9e7185967276bebe6a9c12f3570d821bcb7184e9c703bd278c1fa44ece65c0fb5b0855fe3fca5eb836d3f41b568a230af3156da0440cfaf5d4acd7123506", "expected_verify": false, - "error": "signature_does_not_bind_said" + "error": "signature_invalid" }, { "name": "sig-004-wrong-key", - "description": "Negative: verify a signature against the SAID it was produced for, but with a *different* verifying key. Verification MUST return false. This catches the case where an adversary supplies a valid signature from a key that is not the byte's current owner.", - "envelope_said": "EAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "description": "sig-002's signature over said-001's SAID, presented with another key (the one the all-zero seed derives). The signature is valid for the key that made it, but that is not this key, so verification MUST fail: a valid signature from a key that is not the byte's owner.", + "envelope_said": "E64kxoz2b4kukifl5lbhnndcfos2bqxdjhp5wgx7zh35erpsv243a", + "signed_message_hex": "f715776741e2a8a4157d584ed68c4574b4185c693bfb635ff93efa48be55d736", "signing_key_seed": "0000000000000000000000000000000000000000000000000000000000000000", - "verifying_key": "1111111111111111111111111111111111111111111111111111111111111111", - "signature": "92a009a9f0d4cab8720e820b5f642540a2b27b5416503f8fb3762223ebdb69da085ac1e43e15996e458f3613d0f11d8c387b2eaeb4302aeeb00d291612bb0c00", + "verifying_key": "3b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29", + "signature": "ce9b9e7185967276bebe6a9c12f3570d821bcb7184e9c703bd278c1fa44ece65c0fb5b0855fe3fca5eb836d3f41b568a230af3156da0440cfaf5d4acd7123506", "expected_verify": false, - "error": "verifying_key_does_not_match" + "error": "signature_invalid" }, { "name": "sig-005-malformed-signature", - "description": "Negative: verify a structurally malformed signature (wrong length, in this case 63 bytes instead of 64). Verification MUST return false and the implementation MUST NOT panic or leak timing information distinguishing this case from the cryptographic-failure case.", - "envelope_said": "EAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "description": "sig-002's signature cut to 63 bytes. It is refused as malformed before any cryptography runs, and an implementation MUST NOT panic on it.", + "envelope_said": "E64kxoz2b4kukifl5lbhnndcfos2bqxdjhp5wgx7zh35erpsv243a", + "signed_message_hex": "f715776741e2a8a4157d584ed68c4574b4185c693bfb635ff93efa48be55d736", "signing_key_seed": "4ccd089b28ff96da9db6c346ec114e0f5b8a319f35aba624da8cf6ed4fb8a6fb", - "verifying_key": "3d4017c3e843895a92b70aa74d1b7ebc9c982ccf2ec4968cc0cd55f12af4660c", - "signature": "92a009a9f0d4cab8720e820b5f642540a2b27b5416503f8fb3762223ebdb69da085ac1e43e15996e458f3613d0f11d8c387b2eaeb4302aeeb00d291612bb", + "verifying_key": "3d4017c3e843895a92b70aa74d1b7ebc9c982ccf2ec4968cc0cd55f12af4660c", + "signature": "ce9b9e7185967276bebe6a9c12f3570d821bcb7184e9c703bd278c1fa44ece65c0fb5b0855fe3fca5eb836d3f41b568a230af3156da0440cfaf5d4acd71235", "expected_verify": false, - "error": "malformed_signature_wrong_length" + "error": "malformed_signature" + }, + { + "name": "sig-006-non-canonical-s-rejected", + "description": "sig-002's signature with its scalar S replaced by S + L: the same scalar, written outside [0, L). RFC 8032 §5.1.7 requires refusing it. Accepting it would make signatures malleable, giving a second valid-looking signature for one message and key.", + "envelope_said": "E64kxoz2b4kukifl5lbhnndcfos2bqxdjhp5wgx7zh35erpsv243a", + "signed_message_hex": "f715776741e2a8a4157d584ed68c4574b4185c693bfb635ff93efa48be55d736", + "signing_key_seed": "4ccd089b28ff96da9db6c346ec114e0f5b8a319f35aba624da8cf6ed4fb8a6fb", + "verifying_key": "3d4017c3e843895a92b70aa74d1b7ebc9c982ccf2ec4968cc0cd55f12af4660c", + "signature": "ce9b9e7185967276bebe6a9c12f3570d821bcb7184e9c703bd278c1fa44ece65adcf51656f61522235552e76d315359f230af3156da0440cfaf5d4acd7123516", + "expected_verify": false, + "error": "signature_invalid" } ] diff --git a/smart-byte/smart-byte-rs/Cargo.lock b/smart-byte/smart-byte-rs/Cargo.lock index ebcc32cb5..611532150 100644 --- a/smart-byte/smart-byte-rs/Cargo.lock +++ b/smart-byte/smart-byte-rs/Cargo.lock @@ -5942,6 +5942,16 @@ dependencies = [ "zeroize", ] +[[package]] +name = "smart-byte-conformance" +version = "0.1.0" +dependencies = [ + "hex", + "serde_json", + "smart-byte-core", + "smart-byte-lockstep", +] + [[package]] name = "smart-byte-core" version = "0.1.0" @@ -5950,12 +5960,10 @@ dependencies = [ "chrono", "data-encoding", "ed25519-dalek 2.2.0", - "hex", "rand 0.8.6", "serde", "serde_bytes", "serde_cbor", - "serde_json", "thiserror 2.0.18", "uuid", ] diff --git a/smart-byte/smart-byte-rs/Cargo.toml b/smart-byte/smart-byte-rs/Cargo.toml index c3c19684f..f679da825 100644 --- a/smart-byte/smart-byte-rs/Cargo.toml +++ b/smart-byte/smart-byte-rs/Cargo.toml @@ -23,6 +23,7 @@ members = [ "crates/smart-byte-tendermint", "crates/smart-byte-acdc", "crates/smart-byte-oidc4vc", + "crates/smart-byte-conformance", ] [workspace.package] diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/Cargo.toml b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/Cargo.toml new file mode 100644 index 000000000..0f23acb9f --- /dev/null +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/Cargo.toml @@ -0,0 +1,23 @@ +[package] +name = "smart-byte-conformance" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true +description = "Generates and verifies the Smart Byte conformance vector pack (smart-byte/conformance/v1)." +publish = false + +[dependencies] +smart-byte-core = { workspace = true } +smart-byte-lockstep = { workspace = true } +serde_json = "1" +hex = "0.4" + +[[bin]] +name = "gen-conformance-vectors" +path = "src/bin/gen-conformance-vectors.rs" + +[[bin]] +name = "verify-conformance-vectors" +path = "src/bin/verify-conformance-vectors.rs" diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/bin/gen-conformance-vectors.rs b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/bin/gen-conformance-vectors.rs new file mode 100644 index 000000000..38d7b8416 --- /dev/null +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/bin/gen-conformance-vectors.rs @@ -0,0 +1,67 @@ +//! Write the v1 conformance pack. +//! +//! ```text +//! gen-conformance-vectors [--spec-version v1] [--output-dir DIR] [--check] +//! ``` +//! +//! `--output-dir` defaults to the published pack, `smart-byte/conformance/v1`. +//! `--check` writes nothing and exits non-zero if any file in the directory +//! differs from what the reference implementation generates. + +use std::path::PathBuf; +use std::process::ExitCode; + +fn main() -> ExitCode { + let mut out = smart_byte_conformance::published_pack_dir(); + let mut check = false; + let mut args = std::env::args().skip(1); + while let Some(arg) = args.next() { + match arg.as_str() { + "--spec-version" => match args.next().as_deref() { + Some("v1") => {} + other => { + eprintln!("only spec version v1 has a pack; got {other:?}"); + return ExitCode::from(2); + } + }, + "--output-dir" => match args.next() { + Some(dir) => out = PathBuf::from(dir), + None => { + eprintln!("--output-dir needs a directory"); + return ExitCode::from(2); + } + }, + "--check" => check = true, + other => { + eprintln!("unknown argument {other:?}"); + eprintln!( + "usage: gen-conformance-vectors [--spec-version v1] [--output-dir DIR] [--check]" + ); + return ExitCode::from(2); + } + } + } + + let mut stale = 0; + for (name, contents) in smart_byte_conformance::pack::generate() { + let path = out.join(name); + if check { + if std::fs::read_to_string(&path).ok().as_deref() == Some(contents.as_str()) { + println!("ok {}", path.display()); + } else { + println!("STALE {}", path.display()); + stale += 1; + } + } else if let Err(e) = std::fs::write(&path, contents) { + eprintln!("write {}: {e}", path.display()); + return ExitCode::FAILURE; + } else { + println!("wrote {}", path.display()); + } + } + if stale > 0 { + eprintln!("{stale} file(s) differ from the reference implementation's output"); + return ExitCode::FAILURE; + } + ExitCode::SUCCESS +} diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/bin/verify-conformance-vectors.rs b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/bin/verify-conformance-vectors.rs new file mode 100644 index 000000000..a55ff4038 --- /dev/null +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/bin/verify-conformance-vectors.rs @@ -0,0 +1,40 @@ +//! Check a conformance pack on disk against the reference implementation. +//! +//! ```text +//! verify-conformance-vectors [--pack DIR] +//! ``` +//! +//! `--pack` defaults to the published pack, `smart-byte/conformance/v1`. +//! Exits non-zero if any vector fails. + +use std::path::PathBuf; +use std::process::ExitCode; + +fn main() -> ExitCode { + let mut dir = smart_byte_conformance::published_pack_dir(); + let mut args = std::env::args().skip(1); + while let Some(arg) = args.next() { + match (arg.as_str(), args.next()) { + ("--pack", Some(d)) => dir = PathBuf::from(d), + _ => { + eprintln!("usage: verify-conformance-vectors [--pack DIR]"); + return ExitCode::from(2); + } + } + } + + let report = smart_byte_conformance::verify::verify_pack(&dir); + for c in &report.checks { + match &c.result { + Ok(()) => println!("ok {:<32} {}", c.file, c.vector), + Err(e) => println!("FAIL {:<32} {}: {e}", c.file, c.vector), + } + } + let failed = report.failures().len(); + println!("{} checks, {failed} failed", report.checks.len()); + if report.passed() { + ExitCode::SUCCESS + } else { + ExitCode::FAILURE + } +} diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/gossip.rs b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/gossip.rs new file mode 100644 index 000000000..145845339 --- /dev/null +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/gossip.rs @@ -0,0 +1,71 @@ +//! What a node does with an object the gossip overlay hands it. +//! +//! Spec v1 describes the overlay in prose and defers its propagation rules to +//! a later revision (treatise §21.4, §21.6). What it does fix is enough to +//! decide every delivery the pack's gossip vectors make: +//! +//! * received bytes must be canonical, and are refused rather than re-encoded +//! when they are not (identity spec §2.5); +//! * they must hash to the SAID they state, and that SAID must be the one they +//! were announced or requested under (identity spec §2.1; treatise §10.2 +//! step 1, §21.4); +//! * a SAID already held is a duplicate: the overlay deduplicates by content +//! address (treatise §11.4). +//! +//! No first-come rule is needed. Two different canonical objects cannot share +//! a SAID short of a BLAKE3 collision, so an object that arrives under a held +//! SAID either is the held one or fails to hash to it. + +use smart_byte_core::{Said, said}; + +/// What became of one delivery. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Delivery { + Accepted(Said), + Duplicate(Said), + /// Refused, with the pack's error class. + Rejected(&'static str), +} + +/// A node's store of objects received through the overlay. +#[derive(Debug, Default)] +pub struct Receiver { + held: Vec<(Said, Vec)>, +} + +impl Receiver { + pub fn new() -> Self { + Self::default() + } + + /// Deliver `bytes`, announced (or requested) under `claimed_said`. + pub fn deliver(&mut self, claimed_said: &str, bytes: &[u8]) -> Delivery { + let said = match said::verify(bytes, "d") { + Ok(s) => s, + Err(e) => return Delivery::Rejected(e.class()), + }; + match Said::from_said_text(claimed_said) { + Err(_) => return Delivery::Rejected("malformed_said"), + Ok(claimed) if claimed != said => return Delivery::Rejected("said_mismatch"), + Ok(_) => {} + } + if self.held.iter().any(|(s, _)| *s == said) { + return Delivery::Duplicate(said); + } + self.held.push((said, bytes.to_vec())); + Delivery::Accepted(said) + } + + /// The SAIDs held, in the order they were accepted. + pub fn held(&self) -> Vec { + self.held.iter().map(|(s, _)| *s).collect() + } + + /// The bytes held under `said`: exactly the bytes that were delivered. + pub fn bytes(&self, said: &Said) -> Option<&[u8]> { + self.held + .iter() + .find(|(s, _)| s == said) + .map(|(_, b)| b.as_slice()) + } +} diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/json.rs b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/json.rs new file mode 100644 index 000000000..964d57c3c --- /dev/null +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/json.rs @@ -0,0 +1,252 @@ +//! The pack's JSON: an order-preserving value type with a deterministic +//! printer, and the typed mapping between JSON and the CBOR data model. +//! +//! JSON has no byte-string type, so the pack writes a byte string as an object +//! whose only member is `$bytes`, holding lowercase hex. Every other JSON value +//! maps to CBOR the obvious way: object → map with text keys, array → array, +//! string → text, integer → integer, `true`/`false`/`null` → simple values. +//! A JSON number with a fraction or exponent is refused: floats have no +//! canonical form. + +use smart_byte_core::canonical::Value; + +/// A JSON value whose objects keep the order they were built in, so the +/// generator's output is byte-for-byte reproducible. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum J { + Null, + Bool(bool), + Int(i128), + Str(String), + Arr(Vec), + Obj(Vec<(String, J)>), +} + +impl J { + pub fn str(s: impl Into) -> J { + J::Str(s.into()) + } + + pub fn obj>(entries: impl IntoIterator) -> J { + J::Obj(entries.into_iter().map(|(k, v)| (k.into(), v)).collect()) + } + + pub fn strs>(items: impl IntoIterator) -> J { + J::Arr(items.into_iter().map(|s| J::Str(s.into())).collect()) + } +} + +impl From<&str> for J { + fn from(s: &str) -> J { + J::Str(s.to_owned()) + } +} + +impl From for J { + fn from(s: String) -> J { + J::Str(s) + } +} + +impl From for J { + fn from(b: bool) -> J { + J::Bool(b) + } +} + +impl From for J { + fn from(n: u64) -> J { + J::Int(i128::from(n)) + } +} + +impl From for J { + fn from(n: usize) -> J { + J::Int(n as i128) + } +} + +/// Print `j` with two-space indentation and a trailing newline. Non-ASCII text +/// is written as UTF-8, not escaped. +pub fn pretty(j: &J) -> String { + let mut out = String::new(); + write(j, 0, &mut out); + out.push('\n'); + out +} + +fn write(j: &J, indent: usize, out: &mut String) { + match j { + J::Null => out.push_str("null"), + J::Bool(b) => out.push_str(if *b { "true" } else { "false" }), + J::Int(n) => out.push_str(&n.to_string()), + J::Str(s) => write_str(s, out), + J::Arr(items) if items.is_empty() => out.push_str("[]"), + J::Obj(entries) if entries.is_empty() => out.push_str("{}"), + J::Arr(items) => { + out.push_str("[\n"); + for (i, item) in items.iter().enumerate() { + pad(indent + 2, out); + write(item, indent + 2, out); + out.push_str(if i + 1 < items.len() { ",\n" } else { "\n" }); + } + pad(indent, out); + out.push(']'); + } + J::Obj(entries) => { + out.push_str("{\n"); + for (i, (k, v)) in entries.iter().enumerate() { + pad(indent + 2, out); + write_str(k, out); + out.push_str(": "); + write(v, indent + 2, out); + out.push_str(if i + 1 < entries.len() { ",\n" } else { "\n" }); + } + pad(indent, out); + out.push('}'); + } + } +} + +fn pad(n: usize, out: &mut String) { + out.extend(std::iter::repeat_n(' ', n)); +} + +fn write_str(s: &str, out: &mut String) { + out.push('"'); + for c in s.chars() { + match c { + '"' => out.push_str("\\\""), + '\\' => out.push_str("\\\\"), + '\n' => out.push_str("\\n"), + '\r' => out.push_str("\\r"), + '\t' => out.push_str("\\t"), + c if (c as u32) < 0x20 => out.push_str(&format!("\\u{:04x}", c as u32)), + c => out.push(c), + } + } + out.push('"'); +} + +/// A byte string in the pack's typed JSON. +pub fn bytes_json(b: &[u8]) -> J { + J::obj([("$bytes", J::Str(hex::encode(b)))]) +} + +/// The pack's typed JSON for a CBOR value, keeping map entries in the order +/// the value holds them. +/// +/// # Panics +/// On a map key that is not text and on simple values other than +/// `true`/`false`/`null`: the typed JSON has no spelling for either, and the +/// pack's generator never builds one. +pub fn to_json(v: &Value) -> J { + match v { + Value::Unsigned(n) => J::Int(i128::from(*n)), + Value::Negative(n) => J::Int(-1 - i128::from(*n)), + Value::Bytes(b) => bytes_json(b), + Value::Text(t) => J::Str(t.clone()), + Value::Array(items) => J::Arr(items.iter().map(to_json).collect()), + Value::Map(entries) => J::Obj( + entries + .iter() + .map(|(k, v)| match k { + Value::Text(t) => (t.clone(), to_json(v)), + other => panic!("typed JSON has text keys only, not {other:?}"), + }) + .collect(), + ), + Value::Bool(b) => J::Bool(*b), + Value::Null => J::Null, + Value::Simple(n) => panic!("typed JSON has no spelling for simple value {n}"), + } +} + +/// The CBOR value a typed-JSON value denotes. +pub fn from_json(j: &serde_json::Value) -> Result { + use serde_json::Value as S; + Ok(match j { + S::Null => Value::Null, + S::Bool(b) => Value::Bool(*b), + S::Number(n) => { + if let Some(u) = n.as_u64() { + Value::Unsigned(u) + } else if let Some(i) = n.as_i64() { + Value::int(i) + } else { + return Err(format!( + "{n} is not an integer; floats have no canonical form" + )); + } + } + S::String(s) => Value::Text(s.clone()), + S::Array(items) => Value::Array(items.iter().map(from_json).collect::>()?), + S::Object(map) => { + if map.len() == 1 + && let Some(S::String(h)) = map.get("$bytes") + { + if h.len() % 2 != 0 || h.bytes().any(|c| !matches!(c, b'0'..=b'9' | b'a'..=b'f')) { + return Err(format!( + "$bytes must be lowercase hex of even length: {h:?}" + )); + } + return Ok(Value::Bytes(hex::decode(h).map_err(|e| e.to_string())?)); + } + Value::Map( + map.iter() + .map(|(k, v)| Ok((Value::Text(k.clone()), from_json(v)?))) + .collect::>()?, + ) + } + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn typed_json_round_trips_the_data_model() { + let v = Value::map([ + ("z", Value::int(-5)), + ("bytes", Value::Bytes(vec![0, 0xff])), + ( + "list", + Value::Array(vec![Value::Null, Value::Bool(false), Value::text("t")]), + ), + ("empty", Value::Bytes(vec![])), + ("big", Value::Unsigned(u64::MAX)), + ]); + let printed = pretty(&to_json(&v)); + let parsed: serde_json::Value = serde_json::from_str(&printed).unwrap(); + let back = from_json(&parsed).unwrap(); + // serde_json reorders object members, so compare canonical encodings. + assert_eq!( + smart_byte_core::canonical::encode(&back).unwrap(), + smart_byte_core::canonical::encode(&v).unwrap() + ); + } + + #[test] + fn floats_and_loose_hex_are_refused() { + assert!(from_json(&serde_json::json!(1.5)).is_err()); + assert!(from_json(&serde_json::json!({"$bytes": "ABCD"})).is_err()); + assert!(from_json(&serde_json::json!({"$bytes": "abc"})).is_err()); + // An object with `$bytes` and another member is a map, not bytes. + assert!(matches!( + from_json(&serde_json::json!({"$bytes": "00", "x": 1})).unwrap(), + Value::Map(_) + )); + } + + #[test] + fn the_printer_escapes_only_what_json_requires() { + let j = J::obj([("k", J::str("a\"b\\c\nd — é\u{1}"))]); + assert_eq!( + pretty(&j), + "{\n \"k\": \"a\\\"b\\\\c\\nd — é\\u0001\"\n}\n" + ); + let parsed: serde_json::Value = serde_json::from_str(&pretty(&j)).unwrap(); + assert_eq!(parsed["k"], "a\"b\\c\nd — é\u{1}"); + } +} diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/lib.rs b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/lib.rs new file mode 100644 index 000000000..a03e8cdf0 --- /dev/null +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/lib.rs @@ -0,0 +1,20 @@ +//! The Smart Byte conformance vector pack (`smart-byte/conformance/v1`): +//! generated from the reference implementation by [`pack::generate`], and +//! checked against it by [`verify::verify_pack`]. +//! +//! * `gen-conformance-vectors` writes the pack. +//! * `verify-conformance-vectors` checks a pack on disk, whoever wrote it. +//! * `tests/pack.rs` runs both against the published pack under `cargo test`, +//! so CI fails if the pack and the implementation ever disagree. + +pub mod gossip; +pub mod json; +pub mod pack; +pub mod verify; + +use std::path::PathBuf; + +/// The published v1 pack, beside the spec in this repository. +pub fn published_pack_dir() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../../conformance/v1") +} diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/pack.rs b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/pack.rs new file mode 100644 index 000000000..ac3f76956 --- /dev/null +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/pack.rs @@ -0,0 +1,923 @@ +//! The v1 vector pack, built from the reference implementation. +//! +//! Inputs, names and descriptions are written here; every expected value is +//! computed by `smart-byte-core` and `smart-byte-lockstep` at generation time. +//! A negative vector also states the error class it is meant to produce, and +//! generation fails if the reference refuses it for any other reason — so a +//! vector cannot pass by accident while testing nothing. + +use std::collections::BTreeMap; + +use smart_byte_core::Said; +use smart_byte_core::canonical::{self, Value}; +use smart_byte_core::said::{self, SAID_PLACEHOLDER, SaidDerivation}; +use smart_byte_core::sign::{self, SigningKey}; +use smart_byte_lockstep::{supermajority_threshold, tally}; + +use crate::gossip::{Delivery, Receiver}; +use crate::json::{J, pretty, to_json}; + +/// The pack's vector files, in the order the README lists them. +pub const FILES: [&str; 5] = [ + "canonicalisation_vectors.json", + "said_vectors.json", + "signature_vectors.json", + "lockstep_vectors.json", + "gossip_vectors.json", +]; + +/// The seed of RFC 8032 §7.1 test 1. +pub const RFC8032_TEST1_SEED: &str = + "9d61b19deffd5a60ba844af492ec2cc44449c5697b326919703bac031cae7f60"; +/// The seed of RFC 8032 §7.1 test 2. +pub const RFC8032_TEST2_SEED: &str = + "4ccd089b28ff96da9db6c346ec114e0f5b8a319f35aba624da8cf6ed4fb8a6fb"; + +/// Every file of the pack: (file name, contents). +pub fn generate() -> Vec<(&'static str, String)> { + let saids = SaidVectors::build(); + vec![ + (FILES[0], pretty(&canonicalisation())), + (FILES[1], pretty(&saids.json)), + (FILES[2], pretty(&signatures(&saids))), + (FILES[3], pretty(&lockstep())), + (FILES[4], pretty(&gossip(&saids))), + ] +} + +fn cbor(v: &Value) -> Vec { + canonical::encode(v).expect("every value the pack builds has a canonical form") +} + +fn text(s: &str) -> Value { + Value::text(s) +} + +fn uint(n: u64) -> Value { + Value::Unsigned(n) +} + +fn seed(hex_seed: &str) -> [u8; 32] { + hex::decode(hex_seed).unwrap().try_into().unwrap() +} + +// --------------------------------------------------------------------------- +// canonicalisation_vectors.json +// --------------------------------------------------------------------------- + +fn canon_positive(name: &str, description: &str, input: Value) -> J { + let bytes = cbor(&input); + // The decoder must accept what the encoder wrote, unchanged. + assert_eq!(cbor(&canonical::decode(&bytes).unwrap()), bytes, "{name}"); + J::obj([ + ("name", J::from(name)), + ("description", J::from(description)), + ("input", to_json(&input)), + ("expected_cbor_hex", J::Str(hex::encode(bytes))), + ]) +} + +fn canon_negative(name: &str, description: &str, input: &[u8], class: &str) -> J { + let refused = canonical::decode(input).expect_err(name); + assert_eq!( + refused.class(), + class, + "{name} is refused for the wrong reason" + ); + J::obj([ + ("name", J::from(name)), + ("description", J::from(description)), + ("input_cbor_hex", J::Str(hex::encode(input))), + ("error", J::from(class)), + ]) +} + +fn concat(parts: &[&[u8]]) -> Vec { + parts.concat() +} + +fn canonicalisation() -> J { + let payload = Value::Bytes((0u8..8).collect()); + J::Arr(vec![ + canon_positive( + "canon-001-sorted-map-keys", + "Map keys are written in the bytewise order of their canonical encodings (identity spec §2.3). The input lists the keys out of order; the canonical encoding sorts them.", + Value::map([("z", uint(1)), ("a", uint(2)), ("m", uint(3))]), + ), + canon_positive( + "canon-002-shortest-form-integers", + "Integers use the shortest head that holds them: no leading zeros, no over-wide argument. 0, 23, 24, 255, 256, 65535 and 65536 sit on the boundaries of CBOR's argument widths.", + Value::map([ + ("i0", uint(0)), + ("i23", uint(23)), + ("i24", uint(24)), + ("i255", uint(255)), + ("i256", uint(256)), + ("i65535", uint(65535)), + ("i65536", uint(65536)), + ]), + ), + canon_positive( + "canon-003-no-tags", + "A byte string is written untagged, as major type 2. Wrapping bytes in tag 21 (\"expected base64url conversion\") is tempting, but canonical encoding forbids tags (identity spec §2.3). canon-015 is the tagged form, which a decoder refuses rather than silently stripping the tag.", + Value::map([("payload", payload.clone())]), + ), + canon_negative( + "canon-004-floats-forbidden", + "Floating-point values have no canonical form: identity spec §2.3 forbids them, and a value that would be a float elsewhere is carried as a fixed-point integer. The input is a map holding 1.5 as a half-precision float, and a decoder refuses it.", + &concat(&[&[0xa1], &cbor(&text("exact_half")), &[0xf9, 0x3e, 0x00]]), + "float_forbidden", + ), + canon_positive( + "canon-005-bytes-not-text-for-byte-arrays", + "Byte values are byte strings (major type 2) — never text, and never an array of small integers, which is what a generic serializer writes for a fixed-size byte array by default. The members hold a 32-byte digest, a 32-byte public key and a 64-byte signature. A SAID's text form (identity spec §2.2) is text; the digest it spells is bytes.", + Value::map([ + ("said", Value::Bytes(vec![0x00; 32])), + ("key", Value::Bytes(vec![0x11; 32])), + ("sig", Value::Bytes(vec![0x22; 64])), + ]), + ), + canon_positive( + "canon-006-key-order-follows-encoded-bytes", + "Keys sort by their encoded bytes, not their text (identity spec §2.3). As text \"aa\" comes before \"b\"; encoded, \"b\" (0x61 0x62) comes before \"aa\" (0x62 0x61 0x61), because a string's length is in its first byte. canon-012 is the text-ordered encoding, which a decoder refuses.", + Value::map([("aa", uint(1)), ("b", uint(2))]), + ), + canon_positive( + "canon-007-integer-boundaries-and-negatives", + "Negative integers (major type 1) and the 32-bit/64-bit boundary. -1 is 0x20, and -24 is the last one-byte form; 4294967295 is the largest four-byte argument and 4294967296 the smallest eight-byte one.", + Value::Array( + [ + -1i64, -24, -25, -256, -257, -65536, -65537, 4294967295, 4294967296, + ] + .into_iter() + .map(Value::int) + .collect(), + ), + ), + canon_positive( + "canon-008-string-length-boundary", + "A length uses the shortest head too. A 23-character text string carries its length in its first byte; a 24-character one needs the one-byte argument.", + Value::Array(vec![text(&"a".repeat(23)), text(&"b".repeat(24))]), + ), + canon_positive( + "canon-009-empty-containers-and-simple-values", + "An empty array, map, text string and byte string, and the simple values null, true and false — each a single byte.", + Value::map([ + ("a", Value::Array(vec![])), + ("b", Value::Map(vec![])), + ("c", text("")), + ("d", Value::Bytes(vec![])), + ("e", Value::Null), + ("f", Value::Bool(true)), + ("g", Value::Bool(false)), + ]), + ), + canon_negative( + "canon-010-non-shortest-integer-refused", + "Zero written with a one-byte argument (0x18 0x00). Identity spec §2.3: \"a value of 0 is encoded as 0x00, never as 0x18 0x00\". A decoder refuses it rather than normalising it.", + &[0x18, 0x00], + "non_shortest_integer", + ), + canon_negative( + "canon-011-non-shortest-length-refused", + "The one-character text \"a\" with its length in a one-byte argument (0x78 0x01) rather than in the head. A length has one canonical form too.", + &[0x78, 0x01, 0x61], + "non_shortest_length", + ), + canon_negative( + "canon-012-text-ordered-keys-refused", + "canon-006's map with its keys in text order (\"aa\" before \"b\"). The keys are distinct and the map is well-formed, but the order is not the canonical one, so the encoding is refused.", + &concat(&[ + &[0xa2], + &cbor(&text("aa")), + &[0x01], + &cbor(&text("b")), + &[0x02], + ]), + "unsorted_map_keys", + ), + canon_negative( + "canon-013-duplicate-key-refused", + "A map holding the key \"a\" twice. It has no canonical form, and which of the two values a lenient decoder keeps differs between implementations.", + &concat(&[ + &[0xa2], + &cbor(&text("a")), + &[0x01], + &cbor(&text("a")), + &[0x02], + ]), + "duplicate_map_key", + ), + canon_negative( + "canon-014-indefinite-length-refused", + "An array written with an indefinite length (0x9f … 0xff). Identity spec §2.3 forbids indefinite-length items: the same array has a definite-length encoding.", + &[0x9f, 0x01, 0x02, 0xff], + "indefinite_length", + ), + canon_negative( + "canon-015-tagged-byte-string-refused", + "canon-003's map with its byte string wrapped in tag 21. Tags are forbidden (identity spec §2.3): the decoder refuses the input; it does not strip the tag and accept what is left.", + &concat(&[&[0xa1], &cbor(&text("payload")), &[0xd5], &cbor(&payload)]), + "tag_forbidden", + ), + canon_negative( + "canon-016-single-precision-float-refused", + "A single-precision float (the largest finite one). Refused, as every float is.", + &[0xfa, 0x7f, 0x7f, 0xff, 0xff], + "float_forbidden", + ), + canon_negative( + "canon-017-double-precision-float-refused", + "A double-precision float (the largest finite one). Refused, as every float is.", + &[0xfb, 0x7f, 0xef, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff], + "float_forbidden", + ), + canon_negative( + "canon-018-trailing-bytes-refused", + "Two items back to back. A canonical encoding is exactly one item: bytes after it would sit outside the value a SAID commits to, so the input is refused.", + &[0x01, 0x01], + "trailing_bytes", + ), + canon_negative( + "canon-019-invalid-utf8-refused", + "A text string whose bytes (0xc3 0x28) are not UTF-8. CBOR text strings are UTF-8 (RFC 8949 §3.1), so the input is refused.", + &[0x62, 0xc3, 0x28], + "invalid_utf8", + ), + ]) +} + +// --------------------------------------------------------------------------- +// said_vectors.json +// --------------------------------------------------------------------------- + +/// The SAID vectors, plus each positive vector's derivation for the files +/// that refer to them. +pub struct SaidVectors { + pub json: J, + pub by_name: BTreeMap<&'static str, SaidDerivation>, + /// said-009's input: said-008's object with `n` written non-canonically. + pub non_canonical_copy: Vec, +} + +fn said_positive( + by_name: &mut BTreeMap<&'static str, SaidDerivation>, + name: &'static str, + description: &str, + input: Value, +) -> J { + let d = said::derive(&input, "d").expect(name); + assert_eq!( + said::verify(&d.encoded, "d").as_ref(), + Ok(&d.said), + "{name}" + ); + let json = J::obj([ + ("name", J::from(name)), + ("description", J::from(description)), + ("said_field", J::from("d")), + ("input", to_json(&input)), + ("canonical_cbor_hex", J::Str(hex::encode(&d.hashed))), + ("blake3_hex", J::Str(hex::encode(d.said.as_bytes()))), + ("expected_said", J::Str(d.said.to_said_text())), + ("said_cbor_hex", J::Str(hex::encode(&d.encoded))), + ]); + by_name.insert(name, d); + json +} + +fn said_negative( + name: &str, + description: &str, + input: &[u8], + class: &str, + extra: Vec<(&str, J)>, +) -> J { + let refused = said::verify(input, "d").expect_err(name); + assert_eq!( + refused.class(), + class, + "{name} is refused for the wrong reason" + ); + let mut entries = vec![ + ("name", J::from(name)), + ("description", J::from(description)), + ("said_field", J::from("d")), + ("input_cbor_hex", J::Str(hex::encode(input))), + ("error", J::from(class)), + ]; + entries.extend(extra); + J::obj(entries) +} + +impl SaidVectors { + pub fn build() -> Self { + let mut by_name = BTreeMap::new(); + let placeholder = || text(SAID_PLACEHOLDER); + let key = |b: u8| Value::Bytes(vec![b; 32]); + let filler_signature = |b: u8| Value::Bytes(vec![b; 64]); + let joules = |measured: u64, estimated: u64| { + Value::map([ + ("measured_microjoules", uint(measured)), + ("estimated_microjoules", uint(estimated)), + ]) + }; + let usd = Value::map([ + ("type", text("usd.cargo.v1")), + ("denomination_micros", uint(100_000_000)), + ("issuer_pubkey", key(0x11)), + ]); + let provenance = Value::map([ + ("alg", uint(1)), + ("issuer_pubkey", key(0x11)), + ("issuance_frame", uint(1024)), + ("signature", filler_signature(0x22)), + ]); + let envelope = |provenance: Value, chain: Vec, cargo: Value, cost: Value| { + Value::map([ + ("d", placeholder()), + ("provenance", provenance), + ("ownership_chain", Value::Array(chain)), + ("cargo", cargo), + ("joule_cost", cost), + ]) + }; + + let mut vectors = vec![ + said_positive( + &mut by_name, + "said-001-empty-envelope", + "The smallest envelope-shaped object (treatise §8.1): no provenance, no cargo, no ownership transitions, zero joule cost. It exercises the whole pipeline — placeholder, canonical CBOR, BLAKE3-256, `E` + base32 — on an object with no application data.", + envelope(Value::Null, vec![], Value::Null, joules(0, 0)), + ), + said_positive( + &mut by_name, + "said-002-usd-cargo", + "An envelope-shaped object carrying a USD cargo of 100.00 (100000000 micros) from issuer key 0x11…11. Cargo is opaque to the substrate; here it is a map whose members include a byte-string key.", + envelope(Value::Null, vec![], usd.clone(), joules(12, 15)), + ), + said_positive( + &mut by_name, + "said-003-custom-cargo", + "The same shape carrying a non-money cargo: a temperature reading. The pipeline treats it exactly as it treats said-002 — cargo opacity is the property under test.", + envelope( + Value::Null, + vec![], + Value::map([ + ("type", text("sensor.temperature.v1")), + ("sensor_id", text("rooftop-array-7")), + ("celsius_milli", uint(23475)), + ("reading_frame", uint(184236)), + ]), + joules(8, 10), + ), + ), + said_positive( + &mut by_name, + "said-004-with-provenance", + "An envelope-shaped object with a provenance attestation: algorithm identifier 0x01 (Ed25519), issuer key, issuance frame and signature. The SAID covers the provenance. The key and signature are fixed filler bytes: this vector tests SAID derivation, not signature validity, which signature_vectors.json covers.", + envelope(provenance.clone(), vec![], usd.clone(), joules(14, 18)), + ), + ]; + + // said-005 chains from said-004 (treatise §10.1): the byte's identity is + // its initial history hash, and each step is blake3(previous ‖ payload). + let origin = by_name["said-004-with-provenance"].said; + let transition = |from: u8, to: u8, frame: u64, previous: [u8; 32], signature: u8| { + Value::map([ + ("alg", uint(1)), + ("from_pubkey", key(from)), + ("to_pubkey", key(to)), + ("frame", uint(frame)), + ("previous_history_hash", Value::Bytes(previous.to_vec())), + ("signature", filler_signature(signature)), + ]) + }; + let first = transition(0x11, 0x33, 1050, origin.0, 0x44); + let after_first = Said::hash(&[origin.as_bytes().as_slice(), &cbor(&first)].concat()).0; + let second = transition(0x33, 0x55, 1080, after_first, 0x66); + vectors.push(said_positive( + &mut by_name, + "said-005-with-ownership-transitions", + "said-004's byte after two transfers, as fetched at frame 1100. The first transition's previous_history_hash is the digest of said-004's SAID, because a byte's identity is its initial history hash (treatise §10.1); the second is blake3 of that digest followed by the first transition's canonical encoding, illustrating §10.1's chain step (spec v1 does not pin the payload's serialization). This object's SAID differs from said-004's: a SAID is a function of the whole object, which is why a byte's identity is fixed at its origin (treatise §8.2) rather than recomputed as transitions accrue.", + envelope(provenance, vec![first, second], usd, joules(47, 56)), + )); + + vectors.push(said_positive( + &mut by_name, + "said-006-only-said-field", + "The empty-map case of identity spec §2.5: an object whose only member is its SAID field. It is the smallest object a SAID can be derived over — a truly empty map has nowhere to put one.", + Value::map([("d", placeholder())]), + )); + + let rfc_key = SigningKey::from_bytes(&seed(RFC8032_TEST1_SEED)) + .verifying_key() + .to_bytes(); + let prefixed_key = [&[0x01u8][..], &rfc_key].concat(); + vectors.push(said_positive( + &mut by_name, + "said-007-one-field-per-key", + "The one-field case: identity spec §3.4's per-key object {d, k}, whose SAID is a next-key commitment. k is the algorithm identifier 0x01 (Ed25519) followed by the public key of RFC 8032's first test seed, carried as a byte string.", + Value::map([("d", placeholder()), ("k", Value::Bytes(prefixed_key.clone()))]), + )); + assert_eq!( + said::per_key_said(&prefixed_key), + by_name["said-007-one-field-per-key"].said, + "per_key_said must agree with the vector it implements" + ); + + vectors.push(said_positive( + &mut by_name, + "said-008-spec-worked-example", + "The worked example of identity spec §2.4: {\"d\": placeholder, \"n\": 1}. Its canonical encoding is 61 bytes, and §2.4 prints the digest and SAID given here.", + Value::map([("d", placeholder()), ("n", uint(1))]), + )); + + // The adversarial cases of identity spec §2.5, built on said-008. + let example = &by_name["said-008-spec-worked-example"]; + let canonical_said = example.said.to_said_text(); + // n = 1 is the final byte of the canonical form; spell it 0x18 0x01. + let lax_n = |canonical: &[u8]| { + let mut v = canonical[..canonical.len() - 1].to_vec(); + v.extend_from_slice(&[0x18, 0x01]); + v + }; + let non_canonical_copy = lax_n(&example.encoded); + let hashed_as_received = Said::hash(&lax_n(&example.hashed)).to_said_text(); + vectors.push(said_negative( + "said-009-non-canonical-copy-rejected", + "The adversarial case of identity spec §2.5, aimed at re-encoding. These are said-008's bytes with n written 0x18 0x01 instead of 0x01, still stating said-008's SAID. A verifier that re-encodes before hashing gets said-008's bytes back and accepts, so a second byte string would verify under said-008's SAID. A conforming verifier refuses the input as non-canonical before computing anything.", + &non_canonical_copy, + "non_shortest_integer", + vec![ + ("said_of_received_bytes", J::Str(hashed_as_received)), + ("said_after_reencoding", J::Str(canonical_said.clone())), + ], + )); + + // The same object with its keys reversed, stating the SAID of exactly + // those bytes: self-consistent for a verifier that hashes as received. + let reversed = |d: &Value| { + concat(&[ + &[0xa2], + &cbor(&text("n")), + &[0x01], + &cbor(&text("d")), + &cbor(d), + ]) + }; + let reversed_said = Said::hash(&reversed(&placeholder())).to_said_text(); + let reversed_bytes = reversed(&text(&reversed_said)); + vectors.push(said_negative( + "said-010-self-consistent-non-canonical-rejected", + "The adversarial case aimed the other way. said-008's object with its keys in reverse order, stating the SAID of exactly these bytes (placeholder substituted in place). A verifier that hashes what it received, without checking canonical form, accepts it. A conforming verifier refuses it as non-canonical.", + &reversed_bytes, + "unsorted_map_keys", + vec![ + ("said_of_received_bytes", J::Str(reversed_said)), + ("said_after_reencoding", J::Str(canonical_said)), + ], + )); + + let mut changed = example.encoded.clone(); + *changed.last_mut().unwrap() = 0x02; + vectors.push(said_negative( + "said-011-changed-content-rejected", + "said-008's object with n changed from 1 to 2 and its SAID left as it was. The bytes are canonical and well-formed, but they state a SAID they do not hash to.", + &changed, + "said_mismatch", + vec![], + )); + + SaidVectors { + json: J::Arr(vectors), + by_name, + non_canonical_copy, + } + } + + fn said(&self, name: &str) -> &SaidDerivation { + &self.by_name[name] + } +} + +// --------------------------------------------------------------------------- +// signature_vectors.json +// --------------------------------------------------------------------------- + +#[allow(clippy::too_many_arguments)] +fn signature_vector( + name: &str, + description: &str, + said: &Said, + signing_seed: [u8; 32], + signature: &[u8], + expected_verify: bool, + error: Option<&str>, +) -> J { + let verifying_key = SigningKey::from_bytes(&signing_seed) + .verifying_key() + .to_bytes(); + let result = sign::verify_said_signature(&said.to_said_text(), signature, &verifying_key); + assert_eq!(result.is_ok(), expected_verify, "{name}"); + assert_eq!( + result.err().map(|e| e.class()), + error, + "{name} fails for the wrong reason" + ); + J::obj([ + ("name", J::from(name)), + ("description", J::from(description)), + ("envelope_said", J::Str(said.to_said_text())), + ("signed_message_hex", J::Str(hex::encode(said.as_bytes()))), + ("signing_key_seed", J::Str(hex::encode(signing_seed))), + ("verifying_key", J::Str(hex::encode(verifying_key))), + ("signature", J::Str(hex::encode(signature))), + ("expected_verify", J::from(expected_verify)), + ("error", error.map(J::from).unwrap_or(J::Null)), + ]) +} + +/// The Ed25519 group order L, little-endian. +const ED25519_L: [u8; 32] = [ + 0xed, 0xd3, 0xf5, 0x5c, 0x1a, 0x63, 0x12, 0x58, 0xd6, 0x9c, 0xf7, 0xa2, 0xde, 0xf9, 0xde, 0x14, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, +]; + +/// `signature` with its scalar S replaced by S + L: the same scalar, written +/// outside [0, L). +fn with_s_plus_l(signature: &[u8; 64]) -> [u8; 64] { + let mut out = *signature; + let mut carry = 0u16; + for i in 0..32 { + let v = u16::from(signature[32 + i]) + u16::from(ED25519_L[i]) + carry; + out[32 + i] = v as u8; + carry = v >> 8; + } + assert_eq!(carry, 0, "S + L fits in 32 bytes for any S < L"); + out +} + +fn signatures(saids: &SaidVectors) -> J { + let empty = saids.said("said-001-empty-envelope").said; + let usd = saids.said("said-002-usd-cargo").said; + let custom = saids.said("said-003-custom-cargo").said; + let (seed1, seed2, zero) = ( + seed(RFC8032_TEST1_SEED), + seed(RFC8032_TEST2_SEED), + [0u8; 32], + ); + let sig1 = sign::sign_said(&usd, &SigningKey::from_bytes(&seed1)).to_bytes(); + let sig2 = sign::sign_said(&empty, &SigningKey::from_bytes(&seed2)).to_bytes(); + J::Arr(vec![ + signature_vector( + "sig-001-known-key-known-said", + "Sign said-002's SAID with the key of RFC 8032's first test seed. The message is the 32 digest bytes the SAID spells (signed_message_hex), not its 53-character text. Ed25519 is deterministic, so every implementation produces exactly this signature.", + &usd, + seed1, + &sig1, + true, + None, + ), + signature_vector( + "sig-002-verify-positive", + "A signature over said-001's SAID by the key of RFC 8032's second test seed. It verifies under that key.", + &empty, + seed2, + &sig2, + true, + None, + ), + signature_vector( + "sig-003-tampered-envelope", + "sig-002's signature presented with said-003's SAID: an adversary swaps the envelope and forwards the original signature. The signature is well-formed but binds another SAID, so verification MUST fail.", + &custom, + seed2, + &sig2, + false, + Some("signature_invalid"), + ), + signature_vector( + "sig-004-wrong-key", + "sig-002's signature over said-001's SAID, presented with another key (the one the all-zero seed derives). The signature is valid for the key that made it, but that is not this key, so verification MUST fail: a valid signature from a key that is not the byte's owner.", + &empty, + zero, + &sig2, + false, + Some("signature_invalid"), + ), + signature_vector( + "sig-005-malformed-signature", + "sig-002's signature cut to 63 bytes. It is refused as malformed before any cryptography runs, and an implementation MUST NOT panic on it.", + &empty, + seed2, + &sig2[..63], + false, + Some("malformed_signature"), + ), + signature_vector( + "sig-006-non-canonical-s-rejected", + "sig-002's signature with its scalar S replaced by S + L: the same scalar, written outside [0, L). RFC 8032 §5.1.7 requires refusing it. Accepting it would make signatures malleable, giving a second valid-looking signature for one message and key.", + &empty, + seed2, + &with_s_plus_l(&sig2), + false, + Some("signature_invalid"), + ), + ]) +} + +// --------------------------------------------------------------------------- +// lockstep_vectors.json +// --------------------------------------------------------------------------- + +fn lockstep_vector( + name: &str, + description: &str, + frame: u64, + cluster_size: usize, + reports: &[(&str, u8)], +) -> J { + let reports: Vec<(String, [u8; 32])> = reports + .iter() + .map(|(node, fill)| ((*node).to_owned(), [*fill; 32])) + .collect(); + let t = tally(cluster_size, &reports).expect(name); + + let mut entries = Vec::new(); + for (_, hash) in &reports { + if let Some(entry) = t.dissent_entry(frame, hash) + && !entries.contains(&entry) + { + entries.push(entry); + } + } + // §14.2: every committing node writes the same entry, so there is at + // most one per frame. + assert!( + entries.len() <= 1, + "{name}: committing nodes disagree on the dissent entry" + ); + + J::obj([ + ("name", J::from(name)), + ("description", J::from(description)), + ("node_count", J::from(cluster_size)), + ("frame_number", J::from(frame)), + ( + "node_hashes", + J::obj( + reports + .iter() + .map(|(n, h)| (n.clone(), J::Str(hex::encode(h)))), + ), + ), + ( + "expected_threshold", + J::from(supermajority_threshold(cluster_size)), + ), + ("expected_commit", J::from(t.committed())), + ( + "expected_state_hash", + t.supermajority_hash + .map(|h| J::Str(hex::encode(h))) + .unwrap_or(J::Null), + ), + ( + "expected_outcomes", + J::obj( + reports + .iter() + .map(|(n, h)| (n.clone(), J::from(t.outcome(h).as_str()))), + ), + ), + ( + "expected_dissent", + J::Arr( + entries + .into_iter() + .map(|e| { + J::obj([ + ("frame", J::from(e.frame)), + ("dissenting_peers", J::strs(e.dissenting_peers)), + ("local_hash", J::Str(hex::encode(e.local_hash))), + ( + "supermajority_hash", + J::Str(hex::encode(e.supermajority_hash)), + ), + ]) + }) + .collect(), + ), + ), + ]) +} + +fn lockstep() -> J { + J::Arr(vec![ + lockstep_vector( + "lockstep-001-unanimous-commit", + "All four nodes report the same post-frame state hash. The threshold is floor(2·4/3)+1 = 3, so the frame commits, every node's outcome is a clean commit (treatise §13.3), and nothing is logged as dissent. State hashes in these vectors are opaque 32-byte values compared only for equality: §9.4 does not pin the state's serialization, so the commit rule is tested over the hashes nodes report.", + 184237, + 4, + &[ + ("node_1", 0xaa), + ("node_2", 0xaa), + ("node_3", 0xaa), + ("node_4", 0xaa), + ], + ), + lockstep_vector( + "lockstep-002-supermajority-commit", + "Three of four nodes agree and node_4 reports another hash. The threshold of 3 is met and the frame commits at the majority hash. Nodes 1–3 commit with dissent and write one dissent-log entry naming node_4 (treatise §14.1). node_4's own outcome is minority divergence: it halts, and writes nothing.", + 184238, + 4, + &[ + ("node_1", 0xbb), + ("node_2", 0xbb), + ("node_3", 0xbb), + ("node_4", 0xdd), + ], + ), + lockstep_vector( + "lockstep-003-threshold-not-met", + "No hash is reported by more than two of the four nodes. None reaches the threshold of 3, so the frame does not commit and every node's outcome is a cluster split (§13.3). No dissent is recorded: dissent is divergence from a committed hash (§14.2), and there is none.", + 184239, + 4, + &[ + ("node_1", 0xcc), + ("node_2", 0xcc), + ("node_3", 0xdd), + ("node_4", 0xee), + ], + ), + lockstep_vector( + "lockstep-004-three-nodes-one-liar-split", + "The n = 3 exception (treatise §13.2). The threshold is 3, so one node reporting a different hash leaves 2 of the 3 votes the frame needs. The cluster detects the divergence but cannot tolerate it: no commit, and every node halts with a cluster split.", + 184240, + 3, + &[("node_1", 0x11), ("node_2", 0x11), ("node_3", 0x22)], + ), + lockstep_vector( + "lockstep-005-seven-nodes-two-dissenters", + "Seven nodes, threshold 5: five agree and two report hashes of their own. The frame commits. The dissent-log entry names both dissenters in ascending order, so every honest node writes the same entry (§14.2).", + 184241, + 7, + &[ + ("node_1", 0x5a), + ("node_2", 0x5a), + ("node_3", 0x5a), + ("node_4", 0x5a), + ("node_5", 0x5a), + ("node_6", 0x6b), + ("node_7", 0x7c), + ], + ), + ]) +} + +// --------------------------------------------------------------------------- +// gossip_vectors.json +// --------------------------------------------------------------------------- + +fn gossip_vector( + name: &str, + description: &str, + receiver: &str, + deliveries: &[(&str, String, Vec)], +) -> J { + let mut node = Receiver::new(); + let mut results = Vec::new(); + for (_, claimed, bytes) in deliveries { + let result = node.deliver(claimed, bytes); + if let Delivery::Accepted(s) = &result { + assert_eq!( + node.bytes(s), + Some(bytes.as_slice()), + "{name}: held bytes must be the delivered bytes" + ); + } + results.push(match result { + Delivery::Accepted(s) => J::obj([ + ("outcome", J::from("accepted")), + ("said", J::Str(s.to_said_text())), + ]), + Delivery::Duplicate(s) => J::obj([ + ("outcome", J::from("duplicate")), + ("said", J::Str(s.to_said_text())), + ]), + Delivery::Rejected(class) => { + J::obj([("outcome", J::from("rejected")), ("error", J::from(class))]) + } + }); + } + J::obj([ + ("name", J::from(name)), + ("description", J::from(description)), + ("receiver", J::from(receiver)), + ( + "deliveries", + J::Arr( + deliveries + .iter() + .map(|(from, claimed, bytes)| { + J::obj([ + ("from", J::from(*from)), + ("claimed_said", J::Str(claimed.clone())), + ("cbor_hex", J::Str(hex::encode(bytes))), + ]) + }) + .collect(), + ), + ), + ("expected_results", J::Arr(results)), + ( + "expected_held", + J::strs(node.held().iter().map(Said::to_said_text)), + ), + ]) +} + +fn gossip(saids: &SaidVectors) -> J { + let usd = saids.said("said-002-usd-cargo"); + let custom = saids.said("said-003-custom-cargo"); + let origin = saids.said("said-004-with-provenance"); + let transferred = saids.said("said-005-with-ownership-transitions"); + let example = saids.said("said-008-spec-worked-example"); + + // said-003's object with its reading changed and its SAID left as it was. + let mut altered = custom.object.clone(); + *altered + .get_mut("cargo") + .and_then(|c| c.get_mut("celsius_milli")) + .unwrap() = uint(99999); + let altered = cbor(&altered); + + let text = |d: &SaidDerivation| d.said.to_said_text(); + J::Arr(vec![ + gossip_vector( + "gossip-001-cross-node-propagation", + "An envelope published at node_A reaches node_B. node_B accepts it because its bytes are canonical and hash to the SAID they state, which is the SAID it was announced under, and it holds the bytes unchanged. Gossip carries the object itself, never a re-encoding of it.", + "node_B", + &[("node_A", text(usd), usd.encoded.clone())], + ), + gossip_vector( + "gossip-002-said-conflict-rejected", + "node_C holds said-003's object when a second object arrives under the same SAID: said-003's with its reading changed and its SAID left as it was. Content addressing settles the conflict without a first-come rule: the newcomer does not hash to the SAID it states, so it is refused. Two different canonical objects cannot share a SAID short of a BLAKE3 collision.", + "node_C", + &[ + ("node_A", text(custom), custom.encoded.clone()), + ("node_B", text(custom), altered.clone()), + ], + ), + gossip_vector( + "gossip-003-backfill-answer-verified", + "node_B lacks the object behind said-004's SAID and asks for it by identity: any party holding an identifier can request the rest from any peer (treatise §21.4). node_A answers with said-005's object, which is well-formed and self-consistent but not the one requested, so it is refused. node_D's answer hashes to the requested SAID and is accepted. Spec v1 leaves gossip propagation rules to a later revision (treatise §21.6); what it does fix is that an answer is checked against the identifier that was asked for.", + "node_B", + &[ + ("node_A", text(origin), transferred.encoded.clone()), + ("node_D", text(origin), origin.encoded.clone()), + ], + ), + gossip_vector( + "gossip-004-duplicate-delivery-deduplicated", + "The same object reaches node_B from two peers. The second delivery is a duplicate of a SAID already held and changes nothing: the overlay deduplicates by content address (treatise §11.4).", + "node_B", + &[ + ("node_A", text(usd), usd.encoded.clone()), + ("node_C", text(usd), usd.encoded.clone()), + ], + ), + gossip_vector( + "gossip-005-non-canonical-copy-rejected", + "said-009's bytes reach node_B announced under said-008's SAID, the SAID a re-encoding receiver would compute for them. The bytes are not canonical, so a conforming receiver refuses them (identity spec §2.5). Accepting them would let a second byte string circulate under said-008's SAID.", + "node_B", + &[("node_A", text(example), saids.non_canonical_copy.clone())], + ), + gossip_vector( + "gossip-006-substitute-arriving-first-rejected", + "gossip-002 in the other order: the altered object reaches node_C first and is refused, then said-003's object arrives and is accepted. Arrival order decides nothing; the content does.", + "node_C", + &[ + ("node_B", text(custom), altered), + ("node_A", text(custom), custom.encoded.clone()), + ], + ), + ]) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::json::bytes_json; + + #[test] + fn generation_is_deterministic() { + assert_eq!(generate(), generate()); + } + + #[test] + fn the_typed_json_for_bytes_is_one_member() { + assert_eq!(bytes_json(&[0xab]), J::obj([("$bytes", J::from("ab"))])); + } +} diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/verify.rs b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/verify.rs new file mode 100644 index 000000000..9ce7a2cf1 --- /dev/null +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/verify.rs @@ -0,0 +1,473 @@ +//! Check a vector pack on disk against the reference implementation. +//! +//! This reads only the published JSON: every expected value is recomputed +//! from the vector's own inputs, and every negative vector must be refused +//! with the error class it names. It does not consult the generator, so it +//! checks a pack whoever produced it. + +use std::collections::{BTreeMap, BTreeSet}; +use std::path::Path; + +use serde_json::Value as S; +use smart_byte_core::Said; +use smart_byte_core::canonical::{self, Value}; +use smart_byte_core::said::{self, SAID_PLACEHOLDER}; +use smart_byte_core::sign::{self, SigningKey}; +use smart_byte_lockstep::{supermajority_threshold, tally}; + +use crate::gossip::{Delivery, Receiver}; +use crate::json::from_json; +use crate::pack::FILES; + +/// One vector's result. +#[derive(Debug, Clone)] +pub struct Check { + pub file: String, + pub vector: String, + pub result: Result<(), String>, +} + +/// Every check a pack verification made. +#[derive(Debug, Default)] +pub struct Report { + pub checks: Vec, +} + +impl Report { + pub fn failures(&self) -> Vec<&Check> { + self.checks.iter().filter(|c| c.result.is_err()).collect() + } + + pub fn passed(&self) -> bool { + !self.checks.is_empty() && self.failures().is_empty() + } + + fn record(&mut self, file: &str, vector: &str, result: Result<(), String>) { + self.checks.push(Check { + file: file.to_owned(), + vector: vector.to_owned(), + result, + }); + } +} + +type R = Result<(), String>; + +fn field<'a>(v: &'a S, key: &str) -> Result<&'a S, String> { + v.get(key).ok_or_else(|| format!("missing `{key}`")) +} + +fn text<'a>(v: &'a S, key: &str) -> Result<&'a str, String> { + field(v, key)? + .as_str() + .ok_or_else(|| format!("`{key}` is not a string")) +} + +fn hex_field(v: &S, key: &str) -> Result, String> { + hex::decode(text(v, key)?).map_err(|e| format!("`{key}` is not hex: {e}")) +} + +fn same(what: &str, computed: impl PartialEq + std::fmt::Display, published: &str) -> R { + if computed == published.to_owned() { + Ok(()) + } else { + Err(format!( + "{what}: published {published}, reference computes {computed}" + )) + } +} + +/// Check every file of the pack in `dir`. +pub fn verify_pack(dir: &Path) -> Report { + let mut report = Report::default(); + let mut files: BTreeMap<&str, Vec> = BTreeMap::new(); + + for file in FILES { + let loaded = std::fs::read(dir.join(file)) + .map_err(|e| format!("read: {e}")) + .and_then(|b| serde_json::from_slice::(&b).map_err(|e| format!("parse: {e}"))) + .and_then(|j| match j { + S::Array(items) => Ok(items), + _ => Err("the top-level value must be an array".to_owned()), + }); + match loaded { + Ok(items) => { + report.record(file, "(file)", structure(&items)); + files.insert(file, items); + } + Err(e) => report.record(file, "(file)", Err(e)), + } + } + + let said_texts: BTreeSet = files + .get("said_vectors.json") + .into_iter() + .flatten() + .filter_map(|v| { + v.get("expected_said") + .and_then(S::as_str) + .map(str::to_owned) + }) + .collect(); + + for (file, items) in &files { + for v in items { + let name = v.get("name").and_then(S::as_str).unwrap_or("(unnamed)"); + let result = match *file { + "canonicalisation_vectors.json" => canonicalisation(v), + "said_vectors.json" => said_vector(v), + "signature_vectors.json" => signature(v, &said_texts), + "lockstep_vectors.json" => lockstep(v), + "gossip_vectors.json" => gossip(v), + _ => unreachable!("FILES lists exactly these"), + }; + report.record(file, name, result); + } + } + report +} + +/// Names and descriptions present, names unique, supersessions resolvable. +fn structure(items: &[S]) -> R { + let mut names = BTreeSet::new(); + for v in items { + let name = text(v, "name")?; + text(v, "description").map_err(|e| format!("{name}: {e}"))?; + if !names.insert(name) { + return Err(format!("duplicate vector name {name}")); + } + } + for v in items { + if let Some(target) = v.get("superseded_by").and_then(S::as_str) + && !names.contains(target) + { + return Err(format!( + "{} is superseded by {target}, which is not in the file", + v["name"] + )); + } + } + Ok(()) +} + +fn refused_as(what: &str, class: Result<(), &'static str>, v: &S) -> R { + let want = text(v, "error")?; + match class { + Ok(()) => Err(format!("{what} accepted input it must refuse with {want}")), + Err(got) if got == want => Ok(()), + Err(got) => Err(format!("refused with {got}, but the vector says {want}")), + } +} + +fn canonicalisation(v: &S) -> R { + if let Some(input) = v.get("input") { + let value = from_json(input)?; + let bytes = canonical::encode(&value).map_err(|e| e.to_string())?; + same( + "expected_cbor_hex", + hex::encode(&bytes), + text(v, "expected_cbor_hex")?, + )?; + let back = canonical::decode(&bytes) + .map_err(|e| format!("decoder refuses the encoder's output: {e}"))?; + if canonical::encode(&back).map_err(|e| e.to_string())? != bytes { + return Err("decode then encode does not reproduce the bytes".into()); + } + Ok(()) + } else { + let input = hex_field(v, "input_cbor_hex")?; + refused_as( + "the decoder", + canonical::decode(&input).map(drop).map_err(|e| e.class()), + v, + ) + } +} + +fn said_vector(v: &S) -> R { + let field_name = text(v, "said_field")?; + if let Some(input) = v.get("input") { + let value = from_json(input)?; + if value.get(field_name) != Some(&Value::text(SAID_PLACEHOLDER)) { + return Err(format!( + "`input` must hold the placeholder in `{field_name}` (identity spec §2.1 step 1)" + )); + } + let d = said::derive(&value, field_name).map_err(|e| e.to_string())?; + same( + "canonical_cbor_hex", + hex::encode(&d.hashed), + text(v, "canonical_cbor_hex")?, + )?; + same( + "blake3_hex", + hex::encode(d.said.as_bytes()), + text(v, "blake3_hex")?, + )?; + same( + "expected_said", + d.said.to_said_text(), + text(v, "expected_said")?, + )?; + same( + "said_cbor_hex", + hex::encode(&d.encoded), + text(v, "said_cbor_hex")?, + )?; + let verified = said::verify(&hex_field(v, "said_cbor_hex")?, field_name) + .map_err(|e| format!("said_cbor_hex does not verify: {e}"))?; + same( + "verified SAID", + verified.to_said_text(), + text(v, "expected_said")?, + ) + } else { + let input = hex_field(v, "input_cbor_hex")?; + refused_as( + "SAID verification", + said::verify(&input, field_name) + .map(drop) + .map_err(|e| e.class()), + v, + )?; + if let Some(received) = v.get("said_of_received_bytes").and_then(S::as_str) { + same( + "said_of_received_bytes", + said_of_received_bytes(&input, field_name)?, + received, + )?; + } + if let Some(reencoded) = v.get("said_after_reencoding").and_then(S::as_str) { + let value = lenient_decode(&input)?; + let d = said::derive(&value, field_name).map_err(|e| e.to_string())?; + same("said_after_reencoding", d.said.to_said_text(), reencoded)?; + } + Ok(()) + } +} + +/// The SAID a verifier that skips the canonical check would compute: the +/// received bytes with the stated SAID overwritten in place by the placeholder. +fn said_of_received_bytes(input: &[u8], field_name: &str) -> Result { + let stated = match lenient_decode(input)?.get(field_name) { + Some(Value::Text(t)) => t.clone(), + _ => return Err(format!("no text `{field_name}` to substitute")), + }; + let at = input + .windows(stated.len()) + .position(|w| w == stated.as_bytes()) + .ok_or("the stated SAID is not in the bytes")?; + let mut bytes = input.to_vec(); + bytes[at..at + stated.len()].copy_from_slice(SAID_PLACEHOLDER.as_bytes()); + Ok(Said::hash(&bytes).to_said_text()) +} + +/// A decoder that accepts over-long heads and maps in any order — only so the +/// verifier can show what a lenient implementation would compute. Nothing +/// that decides acceptance uses it. +fn lenient_decode(input: &[u8]) -> Result { + fn item(b: &[u8], pos: &mut usize) -> Result { + let initial = *b.get(*pos).ok_or("truncated")?; + *pos += 1; + let (major, ai) = (initial >> 5, initial & 0x1f); + if major == 7 { + return match ai { + 20 => Ok(Value::Bool(false)), + 21 => Ok(Value::Bool(true)), + 22 => Ok(Value::Null), + _ => Err(format!( + "lenient decoder: unsupported simple/float {initial:#x}" + )), + }; + } + let width = match ai { + 0..=23 => 0, + 24 => 1, + 25 => 2, + 26 => 4, + 27 => 8, + _ => return Err(format!("lenient decoder: unsupported head {initial:#x}")), + }; + let mut arg = if width == 0 { u64::from(ai) } else { 0 }; + for _ in 0..width { + arg = (arg << 8) | u64::from(*b.get(*pos).ok_or("truncated")?); + *pos += 1; + } + let mut take = |n: u64| -> Result, String> { + let end = *pos + usize::try_from(n).map_err(|e| e.to_string())?; + let s = b.get(*pos..end).ok_or("truncated")?.to_vec(); + *pos = end; + Ok(s) + }; + Ok(match major { + 0 => Value::Unsigned(arg), + 1 => Value::Negative(arg), + 2 => Value::Bytes(take(arg)?), + 3 => Value::Text(String::from_utf8(take(arg)?).map_err(|e| e.to_string())?), + 4 => Value::Array((0..arg).map(|_| item(b, pos)).collect::>()?), + 5 => Value::Map( + (0..arg) + .map(|_| Ok((item(b, pos)?, item(b, pos)?))) + .collect::>()?, + ), + _ => return Err(format!("lenient decoder: unsupported major type {major}")), + }) + } + let mut pos = 0; + let v = item(input, &mut pos)?; + if pos != input.len() { + return Err("trailing bytes".into()); + } + Ok(v) +} + +fn signature(v: &S, said_texts: &BTreeSet) -> R { + let said_text = text(v, "envelope_said")?; + if !said_texts.contains(said_text) { + return Err(format!( + "envelope_said {said_text} is not the expected_said of any SAID vector" + )); + } + let said = Said::from_said_text(said_text).map_err(|e| e.to_string())?; + same( + "signed_message_hex", + hex::encode(said.as_bytes()), + text(v, "signed_message_hex")?, + )?; + let seed: [u8; 32] = hex_field(v, "signing_key_seed")? + .try_into() + .map_err(|_| "signing_key_seed is not 32 bytes".to_owned())?; + let key = SigningKey::from_bytes(&seed); + same( + "verifying_key", + hex::encode(key.verifying_key().to_bytes()), + text(v, "verifying_key")?, + )?; + let sig = hex_field(v, "signature")?; + let result = sign::verify_said_signature(said_text, &sig, &hex_field(v, "verifying_key")?); + match field(v, "expected_verify")?.as_bool() { + Some(true) => { + result.map_err(|e| format!("a positive vector fails to verify: {e}"))?; + if !field(v, "error")?.is_null() { + return Err("a positive vector names an error".into()); + } + // Ed25519 is deterministic: the published signature is THE signature. + same( + "signature", + hex::encode(sign::sign_said(&said, &key).to_bytes()), + text(v, "signature")?, + ) + } + Some(false) => refused_as("signature verification", result.map_err(|e| e.class()), v), + None => Err("expected_verify is not a boolean".into()), + } +} + +fn lockstep(v: &S) -> R { + let n = field(v, "node_count")? + .as_u64() + .ok_or("node_count is not an integer")? as usize; + let frame = field(v, "frame_number")? + .as_u64() + .ok_or("frame_number is not an integer")?; + let hashes = field(v, "node_hashes")? + .as_object() + .ok_or("node_hashes is not an object")?; + let mut reports = Vec::new(); + for (node, h) in hashes { + let h: [u8; 32] = h + .as_str() + .and_then(|s| hex::decode(s).ok()) + .and_then(|b| b.try_into().ok()) + .ok_or_else(|| format!("{node}'s hash is not 32 bytes of hex"))?; + reports.push((node.clone(), h)); + } + let t = tally(n, &reports).map_err(|e| e.to_string())?; + + same( + "expected_threshold", + supermajority_threshold(n).to_string(), + &field(v, "expected_threshold")?.to_string(), + )?; + same( + "expected_commit", + t.committed().to_string(), + &field(v, "expected_commit")?.to_string(), + )?; + let state = t.supermajority_hash.map(hex::encode); + if field(v, "expected_state_hash")?.as_str() != state.as_deref() { + return Err(format!("expected_state_hash: reference computes {state:?}")); + } + let outcomes = field(v, "expected_outcomes")? + .as_object() + .ok_or("expected_outcomes is not an object")?; + if outcomes.len() != reports.len() { + return Err("expected_outcomes must name every reporting node".into()); + } + for (node, h) in &reports { + let published = outcomes + .get(node) + .and_then(S::as_str) + .ok_or_else(|| format!("no outcome for {node}"))?; + same( + &format!("{node}'s outcome"), + t.outcome(h).as_str().to_owned(), + published, + )?; + } + let mut entries: Vec = Vec::new(); + for (_, h) in &reports { + if let Some(e) = t.dissent_entry(frame, h) { + let j = serde_json::json!({ + "frame": e.frame, + "dissenting_peers": e.dissenting_peers, + "local_hash": hex::encode(e.local_hash), + "supermajority_hash": hex::encode(e.supermajority_hash), + }); + if !entries.contains(&j) { + entries.push(j); + } + } + } + if field(v, "expected_dissent")? != &S::Array(entries.clone()) { + return Err(format!( + "expected_dissent: reference computes {}", + S::Array(entries) + )); + } + Ok(()) +} + +fn gossip(v: &S) -> R { + let deliveries = field(v, "deliveries")? + .as_array() + .ok_or("deliveries is not an array")?; + let expected = field(v, "expected_results")? + .as_array() + .ok_or("expected_results is not an array")?; + if deliveries.len() != expected.len() { + return Err("one expected result per delivery".into()); + } + let mut node = Receiver::new(); + for (i, (d, want)) in deliveries.iter().zip(expected).enumerate() { + let got = match node.deliver(text(d, "claimed_said")?, &hex_field(d, "cbor_hex")?) { + Delivery::Accepted(s) => { + serde_json::json!({"outcome": "accepted", "said": s.to_said_text()}) + } + Delivery::Duplicate(s) => { + serde_json::json!({"outcome": "duplicate", "said": s.to_said_text()}) + } + Delivery::Rejected(class) => serde_json::json!({"outcome": "rejected", "error": class}), + }; + if &got != want { + return Err(format!( + "delivery {i}: published {want}, reference computes {got}" + )); + } + } + let held: Vec = node.held().iter().map(Said::to_said_text).collect(); + if field(v, "expected_held")? != &serde_json::json!(held) { + return Err(format!("expected_held: reference holds {held:?}")); + } + Ok(()) +} diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/tests/pack.rs b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/tests/pack.rs new file mode 100644 index 000000000..4b6b6547f --- /dev/null +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/tests/pack.rs @@ -0,0 +1,292 @@ +//! The published pack, `smart-byte/conformance/v1`, held to the reference +//! implementation under `cargo test` — so CI fails whenever the two disagree. + +use std::collections::BTreeSet; +use std::path::Path; + +use serde_json::{Value, json}; +use smart_byte_conformance::{pack, published_pack_dir, verify::verify_pack}; + +fn published(file: &str) -> String { + std::fs::read_to_string(published_pack_dir().join(file)) + .unwrap_or_else(|e| panic!("read {file}: {e}")) +} + +fn vectors(file: &str) -> Vec { + serde_json::from_str::(&published(file)) + .unwrap() + .as_array() + .unwrap() + .clone() +} + +fn by_name(file: &str, name: &str) -> Value { + vectors(file) + .into_iter() + .find(|v| v["name"] == name) + .unwrap_or_else(|| panic!("{file} has no vector {name}")) +} + +#[test] +fn every_published_vector_agrees_with_the_reference() { + let report = verify_pack(&published_pack_dir()); + let failures = report.failures(); + assert!(failures.is_empty(), "{failures:#?}"); + // One structure check per file and one check per vector: none skipped. + let total: usize = pack::FILES.iter().map(|f| vectors(f).len()).sum(); + assert_eq!(report.checks.len(), pack::FILES.len() + total); +} + +#[test] +fn the_published_pack_is_exactly_what_the_reference_generates() { + for (file, generated) in pack::generate() { + assert!( + published(file) == generated, + "{file} differs from the reference implementation's output; regenerate with \ + `cargo run -p smart-byte-conformance --bin gen-conformance-vectors`" + ); + } +} + +/// Copy the published pack to `dir`, apply `mutate` to one vector, and return +/// the names of the vectors the verifier then rejects. +fn failures_after(dir: &Path, file: &str, name: &str, mutate: impl Fn(&mut Value)) -> Vec { + std::fs::create_dir_all(dir).unwrap(); + for f in pack::FILES { + let mut items = vectors(f); + if f == file { + let v = items.iter_mut().find(|v| v["name"] == name).expect(name); + mutate(v); + } + std::fs::write(dir.join(f), serde_json::to_string_pretty(&items).unwrap()).unwrap(); + } + verify_pack(dir) + .failures() + .into_iter() + .map(|c| c.vector.clone()) + .collect() +} + +/// Falsification: the verifier is not vacuous. Each planted defect — a wrong +/// value, a wrong error class, a negative vector turned positive — must be +/// caught, and caught on the vector it was planted in. +#[test] +fn the_verifier_catches_a_planted_defect_in_every_file() { + let flip_last = |s: &str| { + let mut s = s.to_owned(); + let last = s.pop().unwrap(); + s.push(if last == '0' { '1' } else { '0' }); + s + }; + type Plant = (&'static str, &'static str, Box); + let plants: Vec = vec![ + ( + "canonicalisation_vectors.json", + "canon-001-sorted-map-keys", + Box::new(move |v| { + v["expected_cbor_hex"] = json!(flip_last(v["expected_cbor_hex"].as_str().unwrap())) + }), + ), + ( + "canonicalisation_vectors.json", + "canon-012-text-ordered-keys-refused", + Box::new(|v| v["error"] = json!("duplicate_map_key")), + ), + ( + "canonicalisation_vectors.json", + "canon-004-floats-forbidden", + Box::new(|v| v["input_cbor_hex"] = json!("a16a65786163745f68616c6601")), + ), + ( + "said_vectors.json", + "said-008-spec-worked-example", + Box::new(move |v| { + v["expected_said"] = json!(flip_last(v["expected_said"].as_str().unwrap())) + }), + ), + ( + "said_vectors.json", + "said-009-non-canonical-copy-rejected", + Box::new(|v| v["error"] = json!("said_mismatch")), + ), + ( + "said_vectors.json", + "said-010-self-consistent-non-canonical-rejected", + Box::new(move |v| { + v["said_of_received_bytes"] = + json!(flip_last(v["said_of_received_bytes"].as_str().unwrap())) + }), + ), + ( + "said_vectors.json", + "said-005-with-ownership-transitions", + Box::new(|v| v["input"]["joule_cost"]["measured_microjoules"] = json!(48)), + ), + ( + "signature_vectors.json", + "sig-001-known-key-known-said", + Box::new(move |v| v["signature"] = json!(flip_last(v["signature"].as_str().unwrap()))), + ), + ( + "signature_vectors.json", + "sig-003-tampered-envelope", + Box::new(|v| { + v["expected_verify"] = json!(true); + v["error"] = Value::Null; + }), + ), + ( + "signature_vectors.json", + "sig-006-non-canonical-s-rejected", + Box::new(|v| v["error"] = json!("malformed_signature")), + ), + ( + "lockstep_vectors.json", + "lockstep-002-supermajority-commit", + Box::new(|v| v["expected_dissent"] = json!([])), + ), + ( + "lockstep_vectors.json", + "lockstep-004-three-nodes-one-liar-split", + Box::new(|v| v["expected_commit"] = json!(true)), + ), + ( + "lockstep_vectors.json", + "lockstep-002-supermajority-commit", + Box::new(|v| v["expected_outcomes"]["node_4"] = json!("commit_with_dissent")), + ), + ( + "gossip_vectors.json", + "gossip-002-said-conflict-rejected", + Box::new(|v| { + v["expected_results"][1] = + json!({"outcome": "accepted", "said": v["expected_results"][0]["said"]}) + }), + ), + ( + "gossip_vectors.json", + "gossip-005-non-canonical-copy-rejected", + Box::new(|v| { + let s = v["deliveries"][0]["claimed_said"].clone(); + v["expected_results"][0] = json!({"outcome": "accepted", "said": s}); + v["expected_held"] = json!([s]); + }), + ), + ]; + let root = Path::new(env!("CARGO_TARGET_TMPDIR")).join("planted"); + for (i, (file, name, mutate)) in plants.iter().enumerate() { + let failed = failures_after(&root.join(i.to_string()), file, name, mutate); + assert_eq!( + failed, + vec![name.to_string()], + "planted defect {i} in {file}/{name}" + ); + } +} + +#[test] +fn the_said_pack_holds_the_suite_the_spec_names() { + // Identity spec §2.5: "the empty-map case, the one-field case, the worked + // example above, several envelope-shaped objects from treatise §8, and one + // adversarial case where a non-canonical CBOR encoding ...". + let keys = |name: &str| -> BTreeSet { + by_name("said_vectors.json", name)["input"] + .as_object() + .unwrap() + .keys() + .cloned() + .collect() + }; + let set = |ks: &[&str]| ks.iter().map(|k| k.to_string()).collect::>(); + assert_eq!(keys("said-006-only-said-field"), set(&["d"])); + assert_eq!(keys("said-007-one-field-per-key"), set(&["d", "k"])); + assert_eq!(keys("said-008-spec-worked-example"), set(&["d", "n"])); + for n in 1..=5 { + let name = vectors("said_vectors.json")[n - 1]["name"] + .as_str() + .unwrap() + .to_owned(); + assert_eq!( + keys(&name), + set(&["d", "provenance", "ownership_chain", "cargo", "joule_cost"]), + "{name} has the five fields of treatise §8.1 plus its SAID" + ); + } + let adversarial: Vec = vectors("said_vectors.json") + .into_iter() + .filter(|v| { + matches!( + v["error"].as_str(), + Some("non_shortest_integer" | "unsorted_map_keys") + ) + }) + .map(|v| v["name"].as_str().unwrap().to_owned()) + .collect(); + assert!( + !adversarial.is_empty(), + "no adversarial non-canonical vector" + ); +} + +#[test] +fn every_canonical_rule_has_a_negative_vector() { + let classes: BTreeSet = vectors("canonicalisation_vectors.json") + .into_iter() + .filter_map(|v| v["error"].as_str().map(str::to_owned)) + .collect(); + for rule in [ + "non_shortest_integer", + "non_shortest_length", + "unsorted_map_keys", + "duplicate_map_key", + "float_forbidden", + "tag_forbidden", + "indefinite_length", + "invalid_utf8", + "trailing_bytes", + ] { + assert!( + classes.contains(rule), + "no canonicalisation vector is refused with {rule}" + ); + } +} + +#[test] +fn the_spec_prints_the_worked_example_the_pack_computes() { + let spec = std::fs::read_to_string( + Path::new(env!("CARGO_MANIFEST_DIR")).join("../../../spec/identity_and_key_rotation.md"), + ) + .unwrap(); + let example = by_name("said_vectors.json", "said-008-spec-worked-example"); + for field in ["blake3_hex", "expected_said", "canonical_cbor_hex"] { + let value = example[field].as_str().unwrap(); + assert!( + spec.contains(value), + "identity spec §2.4 does not print the pack's {field} {value}" + ); + } + assert!( + spec.contains("total 61 bytes"), + "§2.4 must give the encoding's real length" + ); + assert!( + !spec.contains("3f a1 7b 9c"), + "§2.4 still prints the old illustrative digest" + ); +} + +#[test] +fn signatures_sign_saids_the_said_pack_derives() { + let saids: BTreeSet = vectors("said_vectors.json") + .into_iter() + .filter_map(|v| v["expected_said"].as_str().map(str::to_owned)) + .collect(); + for v in vectors("signature_vectors.json") { + assert!( + saids.contains(v["envelope_said"].as_str().unwrap()), + "{}", + v["name"] + ); + } +} diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-core/Cargo.toml b/smart-byte/smart-byte-rs/crates/smart-byte-core/Cargo.toml index 675379f81..5624c2a9d 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-core/Cargo.toml +++ b/smart-byte/smart-byte-rs/crates/smart-byte-core/Cargo.toml @@ -18,7 +18,3 @@ thiserror = { workspace = true } uuid = { workspace = true } data-encoding = { workspace = true } rand = { workspace = true } - -[dev-dependencies] -serde_json = "1" -hex = "0.4" diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-core/tests/pack.rs b/smart-byte/smart-byte-rs/crates/smart-byte-core/tests/pack.rs deleted file mode 100644 index 711db9030..000000000 --- a/smart-byte/smart-byte-rs/crates/smart-byte-core/tests/pack.rs +++ /dev/null @@ -1,262 +0,0 @@ -//! Hold `smart-byte/conformance/v1` to what it actually contains. -//! -//! The pack is published for third parties to certify against, and its gate — -//! `.github/workflows/conformance.yml`, named "smart-byte conformance gate" — runs -//! `jq` over it: valid JSON, top-level array, every entry has `name` and -//! `description`, names unique within a file. Those are well-formedness checks. **No -//! value in the pack has ever been compared against the implementation**, and there -//! is no conformance crate in `smart-byte-rs` that reads it. -//! -//! `conformance/v1/regeneration.md` is candid that the `expected_*` fields are -//! placeholders "not yet cryptographically computed", to be regenerated "when -//! `smart-byte-rs` lands". So this is a known state, not a discovered defect. What -//! has changed is the precondition: `smart-byte-rs` **has** landed — 22 crates, in -//! the `standards-build` matrix — so the mechanical pass that doc describes is now -//! possible and overdue. (Its named entry point, `cargo run --bin -//! gen-conformance-vectors`, does not exist yet.) -//! -//! This file therefore does two different jobs, and says which is which: -//! -//! 1. **Verifies what is real.** The `signing_key_seed` → `verifying_key` -//! derivations are genuine Ed25519 and are checked against the implementation's -//! own re-export. -//! 2. **Pins what is not**, so the placeholder state is visible in test output -//! rather than only in a doc nobody opens. Those tests are written to **FAIL -//! once the pack is regenerated**, which is the intended signal to replace them -//! with real assertions — the same pattern `map-conformance` uses for its JSON -//! round-trip defect. -//! -//! Regenerating the pack is a normative change to a published artifact and is a -//! maintainer's call, so nothing here changes a vector. - -use smart_byte_core::sign::{Signature, SigningKey, Verifier, VerifyingKey}; - -fn pack(name: &str) -> serde_json::Value { - let p = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")) - .join("../../../conformance/v1") - .canonicalize() - .expect("the conformance pack must exist next to the spec") - .join(name); - serde_json::from_slice(&std::fs::read(&p).unwrap_or_else(|e| panic!("read {name}: {e}"))) - .unwrap_or_else(|e| panic!("{name} must parse: {e}")) -} - -fn vectors(name: &str) -> Vec { - pack(name).as_array().expect("a vector file is a JSON array").clone() -} - -/// REAL: every positive vector's seed derives exactly the verifying key it publishes. -/// -/// This is the one cryptographic relationship in the pack that holds today, so it is -/// asserted properly rather than pinned. `sig-004-wrong-key` is the deliberate -/// exception — its key is *meant* not to match, and that is asserted too, so the -/// negative case cannot quietly become a positive one. -#[test] -fn published_signing_seeds_derive_their_published_verifying_keys() { - let mut checked = 0; - for v in vectors("signature_vectors.json") { - let name = v["name"].as_str().expect("vector has a name"); - let (Some(seed), Some(vk_hex)) = - (v["signing_key_seed"].as_str(), v["verifying_key"].as_str()) - else { - continue; - }; - let bytes: [u8; 32] = hex::decode(seed) - .expect("seed is hex") - .try_into() - .expect("an Ed25519 seed is 32 bytes"); - let derived = hex::encode(SigningKey::from_bytes(&bytes).verifying_key().to_bytes()); - - if name == "sig-004-wrong-key" { - assert_ne!( - derived, vk_hex, - "sig-004 is the wrong-key negative case; its key must NOT derive from its seed" - ); - } else { - assert_eq!(derived, vk_hex, "{name}: published verifying key is not this seed's"); - checked += 1; - } - } - assert_eq!(checked, 4, "expected 4 positive signature vectors"); -} - -/// PINNED DEFECT: the published signatures are not signatures over the SAIDs they name. -/// -/// `sig-001`'s description says it signs "a known envelope SAID". It does not: the -/// value is RFC 8032's test signature over the **empty message**, which is proven -/// here by verifying it against `b""`. `sig-002`/`sig-003` carry another RFC 8032 -/// signature, over neither their SAID nor the empty message. -/// -/// **This test is written to fail once the pack is regenerated.** When it does, -/// replace it with the real assertion: each signature verifies over its own -/// `envelope_said`. -#[test] -fn the_published_signatures_do_not_yet_sign_their_stated_saids() { - let vs = vectors("signature_vectors.json"); - - let one = vs.iter().find(|v| v["name"] == "sig-001-known-key-known-said").expect("sig-001"); - let vk = VerifyingKey::from_bytes( - &hex::decode(one["verifying_key"].as_str().unwrap()).unwrap().try_into().unwrap(), - ) - .expect("a valid Ed25519 key"); - let sig = Signature::from_bytes( - &hex::decode(one["signature"].as_str().unwrap()).unwrap().try_into().unwrap(), - ); - let said = one["envelope_said"].as_str().unwrap(); - - assert!( - vk.verify(said.as_bytes(), &sig).is_err(), - "sig-001 now signs its SAID — the pack was regenerated; replace this test with \ - the real assertion that every signature verifies over its envelope_said" - ); - assert!( - vk.verify(b"", &sig).is_ok(), - "sig-001 is no longer RFC 8032's empty-message signature, but still does not sign \ - its SAID — the pack changed into a third state that neither assertion describes" - ); -} - -/// PINNED DEFECT: the SAID vectors are filler, and say so on their face. -/// -/// Every envelope carries the literal string ``, every -/// `expected_said` is a repeated-character pattern rather than a digest, and every -/// `canonical_cbor_hex` is the same 7-byte prefix rather than a complete encoding. -/// -/// **Written to fail once regenerated.** -#[test] -fn the_published_said_vectors_are_still_placeholders() { - let vs = vectors("said_vectors.json"); - assert_eq!(vs.len(), 5, "the SAID vector count changed"); - - for v in &vs { - let name = v["name"].as_str().unwrap(); - assert_eq!( - v["envelope"]["said"].as_str(), - Some(""), - "{name}: the envelope no longer carries the placeholder marker — regenerated?" - ); - - // A real SAID is BLAKE3-256 in base32; filler is one character repeated. - let said = v["expected_said"].as_str().expect("expected_said is a string"); - let body = &said[1..]; - assert!( - body.chars().all(|c| c == body.as_bytes()[0] as char), - "{name}: expected_said is no longer a repeated-character pattern — replace this \ - test with a real `Envelope::compute_said()` comparison" - ); - - assert_eq!( - v["canonical_cbor_hex"].as_str().map(str::len), - Some(14), - "{name}: canonical_cbor_hex is no longer the 7-byte truncated prefix" - ); - } -} - -/// PINNED DEFECT: the pack's own internal-consistency claim does not hold. -/// -/// `regeneration.md` states the placeholders are "internally consistent (the same -/// envelope produces the same SAID across vectors)". The converse fails: two -/// *different* envelopes are given the *same* expected SAID, so even as filler the -/// pack asserts a collision. Any implementation deriving SAIDs from envelopes would -/// have to produce one too. -#[test] -fn two_different_envelopes_are_published_with_the_same_said() { - let vs = vectors("said_vectors.json"); - let four = vs.iter().find(|v| v["name"] == "said-004-with-provenance").expect("said-004"); - let five = - vs.iter().find(|v| v["name"] == "said-005-with-ownership-transitions").expect("said-005"); - - assert_ne!(four["envelope"], five["envelope"], "said-004 and said-005 are the same envelope"); - assert_eq!( - four["expected_said"], five["expected_said"], - "the duplicate SAID is fixed — replace this test with a real uniqueness assertion" - ); -} - -/// PINNED DEFECT: two canonicalisation vectors are not decodable hex. -/// -/// `canon-002` and `canon-003` publish `expected_cbor_hex` strings of **odd** length -/// (109 and 37 characters), which cannot be decoded to bytes at all. A third party -/// following the spec fails before reaching any comparison. -/// -/// **Written to fail once fixed.** -#[test] -fn two_canonicalisation_vectors_publish_undecodable_hex() { - let mut odd: Vec<(String, usize)> = Vec::new(); - for v in vectors("canonicalisation_vectors.json") { - let h = v["expected_cbor_hex"].as_str().expect("expected_cbor_hex is a string"); - if h.len() % 2 == 1 { - odd.push((v["name"].as_str().unwrap().to_string(), h.len())); - } - // Whatever its length, it must at least be hex characters. - assert!( - h.bytes().all(|b| b.is_ascii_hexdigit()), - "{}: expected_cbor_hex contains non-hex characters", - v["name"] - ); - } - odd.sort(); - assert_eq!( - odd, - vec![ - ("canon-002-shortest-form-integers".to_string(), 109), - ("canon-003-no-tags".to_string(), 37), - ], - "the set of undecodable canonicalisation vectors changed — if they were fixed, \ - replace this test with a real canonical-CBOR comparison" - ); -} - -/// PINNED DEFECT: the spec makes a **MUST** out of vectors nothing can pass, and -/// describes a suite the published file does not contain. -/// -/// `spec/identity_and_key_rotation.md` says implementers "MUST pass every vector" -/// in the SAID pack. Every `expected_said` in that pack is a repeated-character -/// placeholder rather than a digest, so the requirement is unsatisfiable as written -/// — a conforming implementation cannot be conforming. -/// -/// The same sentence describes a suite of "the empty-map case, the one-field case, -/// the worked example above, several envelope-shaped objects from treatise §8, and -/// one adversarial case where a non-canonical CBOR encoding produces a different -/// digest (to verify that implementations reject non-canonical input rather than -/// silently re-encoding)". The published file holds five vectors, all -/// envelope-shaped. **The adversarial case is the security-relevant one — it is the -/// only negative test named — and it is absent.** An implementation that silently -/// re-encodes non-canonical input would pass this pack completely. -/// -/// **Written to fail once the pack is regenerated**, which is the signal to replace -/// this with a real assertion that the adversarial vector is present and rejected. -#[test] -fn the_spec_mandates_a_suite_the_published_pack_does_not_contain() { - let spec = std::fs::read_to_string( - std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")) - .join("../../../spec/identity_and_key_rotation.md"), - ) - .expect("the identity spec must sit beside the pack"); - - assert!( - spec.contains("MUST pass every vector"), - "the spec no longer mandates the pack; this pin needs rewriting" - ); - assert!( - spec.contains("non-canonical CBOR encoding produces a different digest"), - "the spec no longer describes an adversarial case; this pin needs rewriting" - ); - - let vs = vectors("said_vectors.json"); - let adversarial: Vec<&str> = vs - .iter() - .filter(|v| { - let hay = format!("{} {}", v["name"], v["description"]).to_lowercase(); - hay.contains("non-canonical") || hay.contains("adversarial") || hay.contains("reject") - }) - .map(|v| v["name"].as_str().unwrap_or("")) - .collect(); - - assert!( - adversarial.is_empty(), - "the adversarial non-canonical vector the spec mandates now exists ({adversarial:?}) — \ - replace this test with one that feeds it non-canonical CBOR and requires rejection" - ); -} diff --git a/smart-byte/spec/identity_and_key_rotation.md b/smart-byte/spec/identity_and_key_rotation.md index 1667d8f0a..67bdf7982 100644 --- a/smart-byte/spec/identity_and_key_rotation.md +++ b/smart-byte/spec/identity_and_key_rotation.md @@ -79,34 +79,40 @@ A2 # map(2) 01 # unsigned(1) ``` -Concatenated: `A2 61 64 78 35` + 53×`23` + `61 6E 01`, total 62 bytes. - -BLAKE3-256 of those 62 bytes is (worked example; verifiers should reproduce against the test vector file referenced in §2.5): +Concatenated: `A2 61 64 78 35` + 53×`23` + `61 6E 01`, total 61 bytes: ``` -b3: 3f a1 7b 9c 5d 2e 88 41 ... (32 bytes) +a2616478352323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323616e01 ``` -The 32-byte digest encoded in lowercase bytewise base32 (no padding) yields 52 characters, prefixed with `E` to produce the SAID: +BLAKE3-256 of those 61 bytes is: ``` -E h7 q xt 4 ... (53 ASCII chars total — full string in conformance vectors) +56813033774b7fb781e0f8547eae6ec3ca24728c8af6ae9dd771d1e0258f6d39 +``` + +The 32-byte digest encoded in lowercase bytewise base32 (no padding) yields 52 characters, prefixed with `E` to produce the 53-character SAID: + +``` +Ek2atam3xjn73papa7bkh5ltoypfci4umrl3k5hoxohi6ajmpnu4q ``` The final object, with the placeholder replaced, is: ``` { - "d": "E<52-char base32 of the BLAKE3-256 digest>", + "d": "Ek2atam3xjn73papa7bkh5ltoypfci4umrl3k5hoxohi6ajmpnu4q", "n": 1 } ``` +These are the values of vector `said-008-spec-worked-example` in [`../conformance/v1/said_vectors.json`](../conformance/v1/said_vectors.json), generated by the reference implementation and reproduced by an independent second implementation (`../conformance/crosscheck/verify_v1.py`). + Any verifier in possession of the object replaces `d` with 53 `#` characters, re-encodes canonically, computes BLAKE3-256, base32-encodes with the `E` prefix, and confirms equality with the asserted `d`. ### 2.5 Test-vector hook -The full set of canonical worked examples — placeholder bytes, canonical CBOR bytes, BLAKE3 digest, final SAID — lives in `../conformance/v1/said_vectors.json` (created in issue #6). Implementers MUST pass every vector. The suite covers the empty-map case, the one-field case, the worked example above, several envelope-shaped objects from treatise §8, and one adversarial case where a non-canonical CBOR encoding produces a different digest (to verify that implementations reject non-canonical input rather than silently re-encoding). +The full set of canonical worked examples — placeholder bytes, canonical CBOR bytes, BLAKE3 digest, final SAID — lives in `../conformance/v1/said_vectors.json`. Implementers MUST pass every vector. The suite covers the empty-map case (`said-006`), the one-field case (`said-007`, a §3.4 per-key object), the worked example above (`said-008`), five envelope-shaped objects from treatise §8 (`said-001` to `said-005`), and two adversarial cases where a non-canonical CBOR encoding produces a different digest (`said-009`, `said-010`), to verify that implementations reject non-canonical input rather than silently re-encoding it or hashing it as received. `said-011` is a canonical object that states a SAID its content does not hash to. The canonical-encoding rules of §2.3 are exercised one by one in `../conformance/v1/canonicalisation_vectors.json`. --- @@ -245,10 +251,12 @@ The Authenticated Chained Data Containers (ACDC) specification extends KERI's cr The Rust reference implementation of this section lives in the `smart-byte-rs` workspace: -- `smart-byte-rs/crates/smart-byte-core/src/said.rs` — SAID computation, placeholder substitution, canonical CBOR encoding, per-key SAID derivation, and the conformance-vector harness referenced in §2.5. -- `smart-byte-rs/crates/smart-byte-core/src/keri.rs` — key-event log: event type definitions, the wrapper that pairs events with signatures, the verifier algorithm in §4, and helpers for emitting `icp`, `rot`, and `ixn` events from an in-memory controller state. +- `smart-byte-rs/crates/smart-byte-core/src/canonical.rs` — the canonical CBOR encoding of §2.3, and a strict decoder that refuses non-canonical input instead of re-encoding it. +- `smart-byte-rs/crates/smart-byte-core/src/said.rs` — SAID derivation and verification (§2.1, §2.2), placeholder substitution, and per-key SAID derivation (§3.4). +- `smart-byte-rs/crates/smart-byte-conformance` — the conformance-vector harness referenced in §2.5. It generates `../conformance/v1` from the two modules above and checks the published pack under `cargo test`. +- `smart-byte-rs/crates/smart-byte-keri-witness` — the key-event log: event type definitions, the wrapper that pairs events with signatures, the verifier algorithm in §4, and helpers for emitting `icp`, `rot`, and `ixn` events. **Its event SAIDs do not follow §2 yet**: it hashes a general-purpose CBOR serialization with a zeroed placeholder, not the canonical encoding with the 53-`#` placeholder, so its identifiers are not interoperable with this section. -If these modules are not yet present in `smart-byte-rs` at the time this section lands, the implementer should treat the function signatures and field-name conventions in §§2–4 as the v1 interface contract and implement against them directly; the reference crate will conform to those signatures when it lands. The conformance test vectors in `../conformance/v1/said_vectors.json` (issue #6) are the authoritative reference, independent of any single language's implementation. +Where the reference crate and this section disagree, this section governs. The conformance test vectors in `../conformance/v1/said_vectors.json` are the authoritative reference, independent of any single language's implementation. --- -- 2.54.0 (Apple Git-157) From 7cc81a72dd743bbbb406f8c4e502dd0dc57119a3 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Wed, 30 Sep 2026 21:33:24 -0400 Subject: [PATCH 08/12] smart-byte-core: harden the codec, key decoding, and receiving MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From an independent review of the previous commits. - canonical: the decoder no longer sizes any allocation from a declared count. Bounding each `with_capacity` by the remaining input did not bound nesting: 128 nested containers multiplied it, so a 1 MiB input held 4–8 GiB. Containers now grow as their items decode. An input with several defects now reports the first in stream order. - canonical: simple values other than false, true and null are refused (`simple_value_forbidden`). The data model has nothing to decode them to; identity spec §2.3 now says so. - sign: a key whose encoded y is not below p is refused as malformed (RFC 8032 §5.1.3). ed25519-dalek alone accepts y = p + 3, a second spelling of a valid large-order point; a test pins that the library accepts it and the check refuses it. - inbox: receiving SAID-addressed objects (identity spec §2.6) in a fixed order: canonical bytes, then the SAID they state, then the SAID they arrived under, and only then deduplication. Verification comes first, so a forgery under a held SAID is reported as a mismatch rather than absorbed as a duplicate. - envelope: `from_cbor` accepts only an envelope's own encoding. The underlying decoder tolerated extra members and indefinite lengths, so one envelope had unboundedly many byte strings — each a fresh gossip message id under the same SAID. Co-Authored-By: Claude Opus 5.5 --- .../crates/smart-byte-conformance/src/json.rs | 6 +- .../crates/smart-byte-core/src/canonical.rs | 81 ++++++----- .../crates/smart-byte-core/src/envelope.rs | 38 ++++- .../crates/smart-byte-core/src/inbox.rs | 133 ++++++++++++++++++ .../crates/smart-byte-core/src/lib.rs | 1 + .../crates/smart-byte-core/src/said.rs | 3 + .../crates/smart-byte-core/src/sign.rs | 45 +++++- 7 files changed, 266 insertions(+), 41 deletions(-) create mode 100644 smart-byte/smart-byte-rs/crates/smart-byte-core/src/inbox.rs diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/json.rs b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/json.rs index 964d57c3c..d25f0c5db 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/json.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/json.rs @@ -137,9 +137,8 @@ pub fn bytes_json(b: &[u8]) -> J { /// the value holds them. /// /// # Panics -/// On a map key that is not text and on simple values other than -/// `true`/`false`/`null`: the typed JSON has no spelling for either, and the -/// pack's generator never builds one. +/// On a map key that is not text: the typed JSON has no spelling for one, and +/// the pack's generator never builds one. pub fn to_json(v: &Value) -> J { match v { Value::Unsigned(n) => J::Int(i128::from(*n)), @@ -158,7 +157,6 @@ pub fn to_json(v: &Value) -> J { ), Value::Bool(b) => J::Bool(*b), Value::Null => J::Null, - Value::Simple(n) => panic!("typed JSON has no spelling for simple value {n}"), } } diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/canonical.rs b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/canonical.rs index 735860f14..414016737 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/canonical.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/canonical.rs @@ -1,7 +1,8 @@ //! Canonical CBOR — the encoding every SAID is computed over. //! //! Identity spec §2.3 fixes one byte form per value: RFC 8949 §4.2.1 core -//! deterministic encoding, with floats, tags and indefinite lengths forbidden. +//! deterministic encoding, with floats, tags, indefinite lengths and simple +//! values other than `false`, `true` and `null` forbidden. //! [`encode`] produces that form from a [`Value`]. [`decode`] accepts **only** //! that form: a strict decoder that refuses non-canonical bytes instead of //! re-encoding them, which is the property the spec's adversarial SAID vector @@ -18,6 +19,7 @@ //! | map keys sorted by their encoded bytes, not their text | `unsorted_map_keys` | //! | no key twice in one map | `duplicate_map_key` | //! | no floats | `float_forbidden` | +//! | no simple values but `false`, `true`, `null` | `simple_value_forbidden` | //! | no tags | `tag_forbidden` | //! | no indefinite-length items | `indefinite_length` | //! | text strings are UTF-8 | `invalid_utf8` | @@ -27,15 +29,22 @@ //! `malformed`. Nesting deeper than [`MAX_DEPTH`] reports `too_deep`. That //! limit is this implementation's guard against unbounded recursion, not a //! canonical-form rule. +//! +//! An input with several defects reports the first one in stream order, so +//! every conforming decoder names the same class for it. +//! +//! The decoder never allocates from a declared count. Arrays and maps grow as +//! their items actually decode, so memory stays proportional to the input +//! however deeply a hostile input nests oversized counts. use std::cmp::Ordering; /// How deeply [`decode`] lets arrays and maps nest before refusing the input. pub const MAX_DEPTH: usize = 128; -/// A value in the CBOR data model that canonical encoding admits. -/// -/// There is no float and no tag variant, because neither has a canonical form. +/// A value in the Smart Byte data model: the CBOR values canonical encoding +/// admits. Floats, tags and other simple values have no variant because +/// identity spec §2.3 forbids them. #[derive(Clone, Debug, PartialEq, Eq)] pub enum Value { /// Major type 0: `0 ..= u64::MAX`. @@ -54,9 +63,6 @@ pub enum Value { Bool(bool), /// Simple value 22. Null, - /// Any other simple value: `0 ..= 19`, `23` (undefined), `32 ..= 255`. - /// `24 ..= 31` have no well-formed encoding (RFC 8949 §3.3). - Simple(u8), } impl Value { @@ -128,6 +134,8 @@ pub enum CanonicalError { TagForbidden, #[error("floating-point values are forbidden")] FloatForbidden, + #[error("simple values other than false, true and null are forbidden")] + SimpleValueForbidden, #[error("map keys are not in canonical order")] UnsortedMapKeys, #[error("a map key appears twice")] @@ -150,6 +158,7 @@ impl CanonicalError { CanonicalError::IndefiniteLength => "indefinite_length", CanonicalError::TagForbidden => "tag_forbidden", CanonicalError::FloatForbidden => "float_forbidden", + CanonicalError::SimpleValueForbidden => "simple_value_forbidden", CanonicalError::UnsortedMapKeys => "unsorted_map_keys", CanonicalError::DuplicateMapKey => "duplicate_map_key", CanonicalError::InvalidUtf8 => "invalid_utf8", @@ -160,9 +169,8 @@ impl CanonicalError { /// Encode `value` canonically. /// -/// Fails only for values that have no canonical form: a map holding the same -/// key twice (`duplicate_map_key`) or a simple value in `24 ..= 31` -/// (`malformed`). +/// Fails only for a value with no canonical form: a map holding the same key +/// twice (`duplicate_map_key`). pub fn encode(value: &Value) -> Result, CanonicalError> { let mut out = Vec::new(); encode_into(value, &mut out)?; @@ -226,14 +234,6 @@ fn encode_into(value: &Value, out: &mut Vec) -> Result<(), CanonicalError> { Value::Bool(false) => out.push(0xf4), Value::Bool(true) => out.push(0xf5), Value::Null => out.push(0xf6), - Value::Simple(n) => match *n { - 0..=23 => out.push(0xe0 | *n), - 24..=31 => return Err(CanonicalError::Malformed), - _ => { - out.push(0xf8); - out.push(*n); - } - }, } Ok(()) } @@ -295,15 +295,12 @@ impl<'a> Decoder<'a> { } } + /// A string length or item count. It never sizes an allocation: strings + /// are sliced from the input, which bounds them, and containers grow item + /// by item, so the first defect in stream order is the one reported. fn length(&mut self, ai: u8) -> Result { let n = self.argument(ai, CanonicalError::NonShortestLength)?; - // A length larger than the input cannot be satisfied; refusing it here - // also keeps an adversarial length from driving a huge allocation. - let n = usize::try_from(n).map_err(|_| CanonicalError::Truncated)?; - if n > self.buf.len() - self.pos { - return Err(CanonicalError::Truncated); - } - Ok(n) + usize::try_from(n).map_err(|_| CanonicalError::Truncated) } fn item(&mut self, depth: usize) -> Result { @@ -316,15 +313,15 @@ impl<'a> Decoder<'a> { if major == 7 { return match ai { - 0..=19 | 23 => Ok(Value::Simple(ai)), 20 => Ok(Value::Bool(false)), 21 => Ok(Value::Bool(true)), 22 => Ok(Value::Null), + 0..=19 | 23 => Err(CanonicalError::SimpleValueForbidden), 24 => match self.byte()? { // RFC 8949 §3.3: a two-byte simple value below 32 is not // well-formed (it would have a one-byte encoding). n if n < 32 => Err(CanonicalError::Malformed), - n => Ok(Value::Simple(n)), + _ => Err(CanonicalError::SimpleValueForbidden), }, 25..=27 => Err(CanonicalError::FloatForbidden), // 28..=30 are reserved; 31 is a "break" with nothing to end. @@ -365,17 +362,15 @@ impl<'a> Decoder<'a> { } 4 => { let n = self.length(ai)?; - let mut items = Vec::with_capacity(n); + let mut items = Vec::new(); for _ in 0..n { items.push(self.item(depth + 1)?); } Ok(Value::Array(items)) } 5 => { - // Every entry is at least two bytes, so `length` bounding the - // count by the remaining input bounds it twice over. let n = self.length(ai)?; - let mut entries = Vec::with_capacity(n); + let mut entries = Vec::new(); let mut previous_key: Option<&'a [u8]> = None; for _ in 0..n { let start = self.pos; @@ -472,10 +467,10 @@ mod tests { ("text", Value::text("ünïcode")), ( "list", - Value::Array(vec![Value::Null, Value::Bool(true), Value::Simple(23)]), + Value::Array(vec![Value::Null, Value::Bool(true), Value::Bool(false)]), ), ("nested", Value::map([("x", Value::int(-1000))])), - ("hi", Value::Simple(255)), + ("big", Value::Unsigned(u64::MAX)), ]); let bytes = encode(&v).unwrap(); let back = decode(&bytes).unwrap(); @@ -510,6 +505,11 @@ mod tests { ("1c", Malformed), ("ff", Malformed), ("f81f", Malformed), + ("f7", SimpleValueForbidden), + ("e0", SimpleValueForbidden), + ("f3", SimpleValueForbidden), + ("f820", SimpleValueForbidden), + ("f8ff", SimpleValueForbidden), ("1f", Malformed), ] { assert_eq!(decode(&hex(h)), Err(want), "{h}"); @@ -526,6 +526,21 @@ mod tests { assert!(decode(&ok).is_ok()); } + #[test] + fn the_first_defect_in_stream_order_is_the_one_reported() { + // An oversized count is only a defect once the input runs out; a + // forbidden item met before that is reported first. + use CanonicalError::*; + for (h, want) in [ + ("90cf", TagForbidden), + ("a9fb76", FloatForbidden), + ("9a00010000d5", TagForbidden), + ("9a0001000001", Truncated), + ] { + assert_eq!(decode(&hex(h)), Err(want), "{h}"); + } + } + #[test] fn a_huge_declared_length_is_refused_without_allocating() { assert_eq!( diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/envelope.rs b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/envelope.rs index c8b40691a..e1c8a5c74 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/envelope.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/envelope.rs @@ -35,6 +35,8 @@ pub enum EnvelopeError { Cbor(String), #[error("envelope SAID mismatch (expected {expected}, computed {computed})")] SaidMismatch { expected: Said, computed: Said }, + #[error("the bytes decode, but are not the envelope's own encoding")] + NotItsEncoding, } impl From for EnvelopeError { @@ -89,9 +91,17 @@ impl Envelope { Ok(serde_cbor::to_vec(self)?) } - /// Decode from canonical CBOR. + /// Decode an envelope, accepting only its own encoding: the bytes must be + /// exactly what [`Envelope::to_cbor`] writes for the decoded envelope. + /// + /// The underlying decoder tolerates extra members, indefinite lengths and + /// over-long heads. Without this check one envelope would have unboundedly + /// many byte strings, each a fresh gossip message id under the same SAID. pub fn from_cbor(bytes: &[u8]) -> Result { let env: Self = serde_cbor::from_slice(bytes)?; + if env.to_cbor()? != bytes { + return Err(EnvelopeError::NotItsEncoding); + } Ok(env) } @@ -156,6 +166,32 @@ mod tests { back.verify_said().unwrap(); } + #[test] + fn an_envelope_decodes_only_from_its_own_encoding() { + let bytes = fixture_envelope().to_cbor().unwrap(); + assert_eq!(bytes[0], 0xa5, "a five-member map"); + + // An extra member the decoder would otherwise ignore. + let mut padded = bytes.clone(); + padded[0] = 0xa6; + padded.extend_from_slice(&[0x61, b'x', 0x00]); + assert!(matches!( + Envelope::from_cbor(&padded), + Err(EnvelopeError::NotItsEncoding) + )); + + // The same map written with an indefinite length. + let mut indefinite = bytes.clone(); + indefinite[0] = 0xbf; + indefinite.push(0xff); + assert!(matches!( + Envelope::from_cbor(&indefinite), + Err(EnvelopeError::NotItsEncoding) + )); + + assert!(Envelope::from_cbor(&bytes).is_ok()); + } + #[test] fn mutation_breaks_said() { let mut env = fixture_envelope(); diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/inbox.rs b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/inbox.rs new file mode 100644 index 000000000..441187917 --- /dev/null +++ b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/inbox.rs @@ -0,0 +1,133 @@ +//! Receiving SAID-addressed objects (identity spec §2.6). +//! +//! An object that arrives over the network — gossiped, or as the answer to a +//! request by SAID — is accepted in four steps, in this order: +//! +//! 1. its bytes are the canonical encoding of the object, or it is refused +//! (never re-encoded); +//! 2. it hashes to the SAID it states in `d`; +//! 3. that SAID is the one it was announced or requested under; +//! 4. only then: a SAID already held makes it a duplicate. +//! +//! Verification comes before deduplication, so a forged object announced +//! under a SAID the node already holds is reported as a mismatch — the +//! substitution attempt is visible — rather than absorbed as a duplicate. No +//! first-come rule is needed: two different canonical objects cannot share a +//! SAID short of a BLAKE3 collision. + +use crate::said::{self, SAID_FIELD, Said, SaidCheckError}; + +/// What became of one delivery. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Delivery { + /// Verified and stored. + Accepted(Said), + /// Verified, and already held. + Duplicate(Said), + /// Refused; [`SaidCheckError::class`] names why. + Rejected(SaidCheckError), +} + +/// A node's store of SAID-addressed objects, in the order it accepted them. +#[derive(Debug, Default)] +pub struct Inbox { + held: Vec<(Said, Vec)>, +} + +impl Inbox { + pub fn new() -> Self { + Self::default() + } + + /// Deliver `bytes`, announced or requested under `claimed_said`. + pub fn deliver(&mut self, claimed_said: &str, bytes: &[u8]) -> Delivery { + let said = match said::verify(bytes, SAID_FIELD) { + Ok(s) => s, + Err(e) => return Delivery::Rejected(e), + }; + let claimed = match Said::from_said_text(claimed_said) { + Ok(c) => c, + Err(_) => return Delivery::Rejected(SaidCheckError::MalformedSaid("claimed".into())), + }; + if claimed != said { + return Delivery::Rejected(SaidCheckError::Mismatch { + stated: claimed.to_said_text(), + computed: said.to_said_text(), + }); + } + if self.get(&said).is_some() { + return Delivery::Duplicate(said); + } + self.held.push((said, bytes.to_vec())); + Delivery::Accepted(said) + } + + /// The objects held, in acceptance order: each SAID with exactly the bytes + /// that were delivered. + pub fn held(&self) -> &[(Said, Vec)] { + &self.held + } + + /// The bytes held under `said`. + pub fn get(&self, said: &Said) -> Option<&[u8]> { + self.held + .iter() + .find(|(s, _)| s == said) + .map(|(_, b)| b.as_slice()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::canonical::Value; + use crate::said::SAID_PLACEHOLDER; + + fn object(n: u64) -> said::SaidDerivation { + said::derive( + &Value::map([ + ("d", Value::text(SAID_PLACEHOLDER)), + ("n", Value::Unsigned(n)), + ]), + SAID_FIELD, + ) + .unwrap() + } + + #[test] + fn verification_comes_before_deduplication() { + let one = object(1); + let mut inbox = Inbox::new(); + let text = one.said.to_said_text(); + assert_eq!( + inbox.deliver(&text, &one.encoded), + Delivery::Accepted(one.said) + ); + assert_eq!( + inbox.deliver(&text, &one.encoded), + Delivery::Duplicate(one.said) + ); + + // n changed to 2, d left as it was, announced under the held SAID: a + // mismatch, not a duplicate. + let mut forged = one.encoded.clone(); + *forged.last_mut().unwrap() = 0x02; + match inbox.deliver(&text, &forged) { + Delivery::Rejected(e) => assert_eq!(e.class(), "said_mismatch"), + other => panic!("forged object was {other:?}"), + } + assert_eq!(inbox.held().len(), 1); + assert_eq!(inbox.get(&one.said), Some(one.encoded.as_slice())); + } + + #[test] + fn a_genuine_object_under_another_said_is_refused() { + let (one, two) = (object(1), object(2)); + let mut inbox = Inbox::new(); + match inbox.deliver(&one.said.to_said_text(), &two.encoded) { + Delivery::Rejected(e) => assert_eq!(e.class(), "said_mismatch"), + other => panic!("{other:?}"), + } + assert!(inbox.held().is_empty()); + } +} diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/lib.rs b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/lib.rs index a0a4a92a6..a220b17f8 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/lib.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/lib.rs @@ -26,6 +26,7 @@ pub mod canonical; pub mod cargo; pub mod envelope; +pub mod inbox; pub mod joule_cost; pub mod ownership; pub mod provenance; diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/said.rs b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/said.rs index dff7244e7..2177d9b09 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/said.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/said.rs @@ -32,6 +32,9 @@ pub const BLAKE3_256_CODE: char = 'E'; /// Characters in a SAID's text form: the code plus 52 base32 characters. pub const SAID_TEXT_LEN: usize = 53; +/// The member that carries an object's SAID (identity spec §2.1). +pub const SAID_FIELD: &str = "d"; + /// What a SAID field holds while its object is hashed: `#` 53 times (§2.2). pub const SAID_PLACEHOLDER: &str = "#####################################################"; diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/sign.rs b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/sign.rs index 33a78b1f6..c75a70ef0 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-core/src/sign.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-core/src/sign.rs @@ -68,9 +68,14 @@ impl SaidSignatureError { /// Verify `signature` over the SAID whose identity-spec text form is /// `said_text`, under `verifying_key`. /// -/// Verification is RFC 8032 Ed25519, strict: a signature scalar `S` outside -/// `[0, L)` is refused (RFC 8032 §5.1.7), and so are small-order keys and -/// commitments, which plain verification would let a weak key satisfy. +/// Verification is RFC 8032 Ed25519, strict throughout: +/// +/// * a key whose encoded `y` is not below the field prime is refused before +/// anything else (RFC 8032 §5.1.3), so one key has one encoding; +/// * a signature scalar `S` outside `[0, L)` is refused (RFC 8032 §5.1.7); +/// * small-order keys and commitments are refused, which plain verification +/// would let a weak key satisfy. +/// /// Structural problems are reported before any cryptography runs, so a short /// signature is `malformed_signature` rather than a verification failure. pub fn verify_said_signature( @@ -85,12 +90,23 @@ pub fn verify_said_signature( let key: [u8; 32] = verifying_key .try_into() .map_err(|_| SaidSignatureError::MalformedVerifyingKey)?; + if !encodes_y_below_p(&key) { + return Err(SaidSignatureError::MalformedVerifyingKey); + } let key = VerifyingKey::from_bytes(&key).map_err(|_| SaidSignatureError::MalformedVerifyingKey)?; key.verify_strict(said.as_bytes(), &Signature::from_bytes(&sig)) .map_err(|_| SaidSignatureError::SignatureInvalid) } +/// Whether a compressed Edwards point's `y` (its low 255 bits) is below the +/// field prime `p = 2^255 - 19`. Decoders that skip this accept nineteen +/// out-of-range spellings, some of which decompress to valid points. +fn encodes_y_below_p(point: &[u8; 32]) -> bool { + let top = point[31] & 0x7f; + !(top == 0x7f && point[1..31].iter().all(|&b| b == 0xff) && point[0] >= 0xed) +} + #[cfg(test)] mod tests { use super::*; @@ -145,6 +161,29 @@ mod tests { ); } + #[test] + fn a_key_spelled_with_y_at_or_above_p_is_refused() { + // y = p + 3 (sign bit clear) is an out-of-range spelling of y = 3, + // which is on the curve at large order. The Ed25519 library alone + // accepts it as a key; the range check is what refuses it. + let mut key = [0xffu8; 32]; + key[0] = 0xf0; + key[31] = 0x7f; + assert!(VerifyingKey::from_bytes(&key).is_ok()); + let said = Said::hash(b"k"); + assert_eq!( + verify_said_signature(&said.to_said_text(), &[0u8; 64], &key), + Err(SaidSignatureError::MalformedVerifyingKey) + ); + // The largest in-range y is accepted as an encoding (and then fails + // as a key or a signature, not as a spelling). + let mut edge = [0xffu8; 32]; + edge[0] = 0xec; + edge[31] = 0x7f; + assert!(encodes_y_below_p(&edge)); + assert!(!encodes_y_below_p(&key)); + } + #[test] fn the_same_scalar_spelled_s_plus_l_is_refused() { let key = SigningKey::from_bytes(&[3u8; 32]); -- 2.54.0 (Apple Git-157) From a36226a5940935642af8bf5c78fcd3bee7de373d Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Wed, 30 Sep 2026 21:33:24 -0400 Subject: [PATCH 09/12] smart-byte-lockstep: a node's own view of a frame MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `node_view` tallies a node's peers' reported hashes plus the hash the node computed itself. For a node that reports what it computes, that is the shared tally. For treatise §13.4's transport-level liar — honest state, garbage on the wire — it is not: the liar sees unanimity and commits, while its peers log it as a dissenter. The previous model used one global view, so it could not express the case the treatise calls the substrate's real misbehaviour signal. Tests pin both shapes: the liar logged by its peers at n = 4, and at n = 3 the liar alone believing the frame committed while the honest pair halts. Co-Authored-By: Claude Opus 5.5 --- .../crates/smart-byte-lockstep/src/lib.rs | 57 +++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-lockstep/src/lib.rs b/smart-byte/smart-byte-rs/crates/smart-byte-lockstep/src/lib.rs index 15c0f34ea..9719cb2a4 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-lockstep/src/lib.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-lockstep/src/lib.rs @@ -155,6 +155,30 @@ pub fn tally( }) } +/// One node's view of a frame (treatise §13.3, §13.4): the hashes its peers +/// reported, plus the hash it computed itself in place of whatever it +/// broadcast. +/// +/// For a node that reports what it computes the view is the shared tally. For +/// the transport-level liar of §13.4 — honest internal state, garbage on the +/// wire — it is not: the liar tallies its own honest hash, sees agreement, and +/// commits, while its peers record it as a dissenter. Its misbehaviour surfaces +/// only in their dissent logs, which is exactly the treatise's point. +pub fn node_view( + cluster_size: usize, + reports: &[(P, Blake3Hash)], + node: &P, + computed: &Blake3Hash, +) -> Result, TallyError> { + let mut view: Vec<(P, Blake3Hash)> = reports + .iter() + .filter(|(peer, _)| peer != node) + .cloned() + .collect(); + view.push((node.clone(), *computed)); + tally(cluster_size, &view) +} + impl Tally

{ /// Whether the frame commits. pub fn committed(&self) -> bool { @@ -463,6 +487,39 @@ mod tests { assert!(!tally(5, &[(1, a), (2, a), (3, a)]).unwrap().committed()); } + #[test] + fn a_transport_liar_commits_in_its_own_view_and_is_logged_in_its_peers() { + // §13.4: node 4 computed the honest hash but broadcast another. + let (honest, lie) = ([0xbb; 32], [0xdd; 32]); + let reports = [(1, honest), (2, honest), (3, honest), (4, lie)]; + let peers = tally(4, &reports).unwrap(); + assert_eq!(peers.outcome(&honest), FrameOutcome::CommitWithDissent); + assert_eq!(peers.dissenters, vec![4]); + let own = node_view(4, &reports, &4, &honest).unwrap(); + assert_eq!(own.outcome(&honest), FrameOutcome::CleanCommit); + assert!(own.dissent_entry(1, &honest).is_none()); + } + + #[test] + fn at_three_nodes_the_liar_alone_believes_the_frame_committed() { + // §13.2 + §13.4: the honest pair sees 2-vs-1 and halts; the liar sees + // its own honest hash agree with both peers. + let (honest, lie) = ([0x11; 32], [0x22; 32]); + let reports = [(1, honest), (2, honest), (3, lie)]; + assert_eq!( + node_view(3, &reports, &1, &honest) + .unwrap() + .outcome(&honest), + FrameOutcome::ClusterSplit + ); + assert_eq!( + node_view(3, &reports, &3, &honest) + .unwrap() + .outcome(&honest), + FrameOutcome::CleanCommit + ); + } + #[test] fn malformed_report_sets_are_refused() { let a = [7; 32]; -- 2.54.0 (Apple Git-157) From 58d1efd3536b29be5ab56f766feef0eb72221ff9 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Wed, 30 Sep 2026 21:33:24 -0400 Subject: [PATCH 10/12] smart-byte-libp2p: validate gossip before it is forwarded MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous fix checked envelopes on local delivery only. Gossipsub forwards every message before application code runs, so a forged envelope still crossed every honest node. A three-node test (A — B — C, A and C unconnected) reproduced it: C received the forgery, relayed by B. The default gossipsub config now sets `validate_messages()`, so a received message is held until the application rules on it. `settle_gossip` accepts and forwards a genuine envelope and rejects anything else, which also scores the sender down. Every loop that drives the swarm — startup, `run`, `subscribe`, `next_envelope_on`, `fetch_by_said`, the provider lookup — settles each message it sees, so nothing stalls and nothing forged is relayed. `next_envelope_on` now skips a bad message instead of ending the wait. `tests/gossip_validation.rs` runs A — B — C with B on the crate's own event loop. A publishes a forgery, a padded copy of a genuine envelope, and the genuine one; only the last reaches C. Falsified both ways: without `validate_messages()`, B relays the forgery; without `from_cbor`'s own-encoding check, B relays the padded copy. Co-Authored-By: Claude Opus 5.5 --- .../crates/smart-byte-libp2p/src/behaviour.rs | 6 +- .../crates/smart-byte-libp2p/src/event.rs | 38 ++--- .../crates/smart-byte-libp2p/src/node.rs | 9 +- .../crates/smart-byte-libp2p/src/transport.rs | 114 +++++++++++--- .../tests/gossip_validation.rs | 145 ++++++++++++++++++ 5 files changed, 266 insertions(+), 46 deletions(-) create mode 100644 smart-byte/smart-byte-rs/crates/smart-byte-libp2p/tests/gossip_validation.rs diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/behaviour.rs b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/behaviour.rs index 645a65c1c..611b855b2 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/behaviour.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/behaviour.rs @@ -146,12 +146,16 @@ impl SmartByteBehaviour { } /// Build a sensible default Gossipsub config: signed messages, IDs from -/// content (so identical envelopes deduplicate), short heartbeat. +/// content (so identical envelopes deduplicate), short heartbeat, and +/// application validation: a received message is forwarded only after +/// [`crate::transport::settle_gossip`] accepts it as a genuine envelope, so a +/// forged one stops at the first honest node instead of crossing the mesh. pub fn default_gossipsub_config() -> gossipsub::Config { use std::time::Duration; gossipsub::ConfigBuilder::default() .heartbeat_interval(Duration::from_secs(1)) .validation_mode(gossipsub::ValidationMode::Strict) + .validate_messages() .message_id_fn(|m: &gossipsub::Message| { let h = blake3_hash_message(m); gossipsub::MessageId::from(h.to_vec()) diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/event.rs b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/event.rs index d8bcb6721..5293e34ef 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/event.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/event.rs @@ -42,7 +42,26 @@ pub async fn run(node: &mut Libp2pNode, tx: mpsc::Sender) -> Result<( loop { tokio::select! { ev = node.swarm.select_next_some() => { - if let Some(out) = translate(ev) + let out = match ev { + // Every gossip message is settled here before anything + // else: accepted (and forwarded) only if it is a genuine + // envelope, rejected otherwise. + SwarmEvent::Behaviour(SmartByteBehaviourEvent::Gossipsub( + gossipsub::Event::Message { propagation_source, message_id, message }, + )) => crate::transport::settle_gossip( + &mut node.swarm, + &message_id, + &propagation_source, + &message.data, + ) + .map(|envelope| NodeEvent::EnvelopeReceived { + envelope: Box::new(envelope), + from: propagation_source, + topic: message.topic.into_string(), + }), + other => translate(other), + }; + if let Some(out) = out && tx.send(out).await.is_err() { return Err(Error::ControlClosed); @@ -59,23 +78,6 @@ fn translate(ev: SwarmEvent) -> Option { match ev { SwarmEvent::ConnectionEstablished { peer_id, .. } => Some(NodeEvent::PeerConnected(peer_id)), SwarmEvent::ConnectionClosed { peer_id, .. } => Some(NodeEvent::PeerDisconnected(peer_id)), - SwarmEvent::Behaviour(SmartByteBehaviourEvent::Gossipsub( - gossipsub::Event::Message { - propagation_source, - message, - .. - }, - )) => match crate::transport::accept_envelope(&message.data) { - Ok(envelope) => Some(NodeEvent::EnvelopeReceived { - envelope: Box::new(envelope), - from: propagation_source, - topic: message.topic.into_string(), - }), - Err(e) => { - tracing::warn!(error = %e, "dropped a malformed or mis-addressed envelope from gossipsub"); - None - } - }, SwarmEvent::Behaviour(SmartByteBehaviourEvent::Mdns(mdns::Event::Discovered(peers))) => { // Surface the first newly-discovered peer; additional peers // will be returned through subsequent mDNS events. diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/node.rs b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/node.rs index 16cad3393..761583e2a 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/node.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/node.rs @@ -125,10 +125,13 @@ impl Libp2pNode { { let remaining = deadline.saturating_duration_since(tokio::time::Instant::now()); match tokio::time::timeout(remaining, swarm.select_next_some()).await { - Ok(SwarmEvent::NewListenAddr { address, .. }) => { - listen_addrs.push(address); + Ok(ev) => { + if let Some(SwarmEvent::NewListenAddr { address, .. }) = + crate::transport::settle_if_gossip(&mut swarm, ev) + { + listen_addrs.push(address); + } } - Ok(_) => {} Err(_) => break, } } diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/transport.rs b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/transport.rs index 549b48a45..baad08ba8 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/transport.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/src/transport.rs @@ -28,23 +28,20 @@ use std::time::Duration; use libp2p::futures::{Stream, StreamExt}; -use libp2p::gossipsub::{IdentTopic, MessageId}; +use libp2p::gossipsub::{self, IdentTopic, MessageAcceptance, MessageId}; use libp2p::request_response::{self, OutboundRequestId}; use libp2p::swarm::SwarmEvent; -use libp2p::{PeerId, kad}; +use libp2p::{PeerId, Swarm, kad}; use smart_byte_core::{Envelope, Said}; -use crate::behaviour::SmartByteBehaviourEvent; +use crate::behaviour::{SmartByteBehaviour, SmartByteBehaviourEvent}; use crate::codec::{EnvelopeRequest, EnvelopeResponse}; use crate::error::{Error, Result}; use crate::node::Libp2pNode; -/// Decode an envelope received from the network, accepting it only if its -/// content hashes to the id it states (treatise §10.2 step 1). -/// -/// Every gossip receive path goes through here. Before it did, none of them -/// checked, so an envelope stating any id was delivered under that id: a -/// substituted envelope propagated exactly like the real one. +/// Decode an envelope received from the network, accepting it only if the +/// bytes are its own encoding and its content hashes to the id it states +/// (treatise §10.2 step 1). pub(crate) fn accept_envelope( bytes: &[u8], ) -> std::result::Result { @@ -53,6 +50,54 @@ pub(crate) fn accept_envelope( Ok(env) } +/// Rule on one received gossip message. +/// +/// [`crate::behaviour::default_gossipsub_config`] holds every received message +/// unforwarded until the application rules on it. This accepts — and so +/// forwards — only a genuine envelope, and rejects anything else, which also +/// counts against the peer that sent it. Every loop that drives the swarm +/// settles each message it sees, so nothing stalls and nothing forged is +/// relayed. Returns the envelope when it was accepted. +pub fn settle_gossip( + swarm: &mut Swarm, + message_id: &MessageId, + propagation_source: &PeerId, + data: &[u8], +) -> Option { + let verdict = accept_envelope(data); + let acceptance = match &verdict { + Ok(_) => MessageAcceptance::Accept, + Err(e) => { + tracing::warn!(error = %e, peer = %propagation_source, "rejected a gossiped message"); + MessageAcceptance::Reject + } + }; + swarm + .behaviour_mut() + .gossipsub + .report_message_validation_result(message_id, propagation_source, acceptance); + verdict.ok() +} + +/// Settle `ev` if it is a gossip message, for loops waiting on something +/// else; pass every other event through. +pub(crate) fn settle_if_gossip( + swarm: &mut Swarm, + ev: SwarmEvent, +) -> Option> { + match ev { + SwarmEvent::Behaviour(SmartByteBehaviourEvent::Gossipsub(gossipsub::Event::Message { + propagation_source, + message_id, + message, + })) => { + settle_gossip(swarm, &message_id, &propagation_source, &message.data); + None + } + other => Some(other), + } +} + /// Accept a fetch's answer only if it is the envelope that was asked for: it /// must state the requested SAID, and its content must hash to it. Otherwise a /// peer could answer any request with any envelope it liked. @@ -127,12 +172,17 @@ pub async fn subscribe<'a>( if let SwarmEvent::Behaviour(SmartByteBehaviourEvent::Gossipsub( libp2p::gossipsub::Event::Message { propagation_source, + message_id, message, - .. }, )) = ev + && let Some(env) = settle_gossip( + &mut node.swarm, + &message_id, + &propagation_source, + &message.data, + ) && message.topic.as_str() == topic_owned - && let Ok(env) = accept_envelope(&message.data) { return Some(( (env, propagation_source), @@ -145,8 +195,9 @@ pub async fn subscribe<'a>( Ok(Box::pin(stream)) } -/// Poll the swarm until the next gossipsub envelope arrives on -/// `topic`, or `timeout` elapses. +/// Poll the swarm until the next genuine gossipsub envelope arrives on +/// `topic`, or `timeout` elapses. A message that is not a genuine envelope is +/// rejected and skipped: one bad peer cannot end the wait. pub async fn next_envelope_on( node: &mut Libp2pNode, topic: &str, @@ -164,12 +215,21 @@ pub async fn next_envelope_on( Ok(SwarmEvent::Behaviour(SmartByteBehaviourEvent::Gossipsub( libp2p::gossipsub::Event::Message { propagation_source, + message_id, message, - .. }, - ))) if message.topic.as_str() == topic => { - let env = accept_envelope(&message.data).map_err(Error::Envelope)?; - return Ok((env, propagation_source)); + ))) => { + let env = settle_gossip( + &mut node.swarm, + &message_id, + &propagation_source, + &message.data, + ); + if let Some(env) = env + && message.topic.as_str() == topic + { + return Ok((env, propagation_source)); + } } Ok(_) => {} Err(_) => { @@ -234,6 +294,9 @@ async fn direct_fetch(node: &mut Libp2pNode, peer: PeerId, said: Said) -> Result } match tokio::time::timeout(remaining, node.swarm.select_next_some()).await { Ok(ev) => { + let Some(ev) = settle_if_gossip(&mut node.swarm, ev) else { + continue; + }; if let Some(envelope) = handle_event_for_response(ev, request_id, peer)? { return check_served(peer, said, envelope); } @@ -297,8 +360,14 @@ async fn find_provider(node: &mut Libp2pNode, said: Said) -> Result { if remaining.is_zero() { return Err(Error::Kademlia("provider lookup timed out".to_string())); } - match tokio::time::timeout(remaining, node.swarm.select_next_some()).await { - Ok(SwarmEvent::Behaviour(SmartByteBehaviourEvent::Kad( + let ev = match tokio::time::timeout(remaining, node.swarm.select_next_some()).await { + Ok(ev) => settle_if_gossip(&mut node.swarm, ev), + Err(_) => { + return Err(Error::Kademlia("provider lookup timed out".to_string())); + } + }; + match ev { + Some(SwarmEvent::Behaviour(SmartByteBehaviourEvent::Kad( kad::Event::OutboundQueryProgressed { id, result: kad::QueryResult::GetProviders(Ok(progress)), @@ -314,7 +383,7 @@ async fn find_provider(node: &mut Libp2pNode, said: Said) -> Result { return Err(Error::Kademlia("no providers found".to_string())); } }, - Ok(SwarmEvent::Behaviour(SmartByteBehaviourEvent::Kad( + Some(SwarmEvent::Behaviour(SmartByteBehaviourEvent::Kad( kad::Event::OutboundQueryProgressed { id, result: kad::QueryResult::GetProviders(Err(e)), @@ -323,10 +392,7 @@ async fn find_provider(node: &mut Libp2pNode, said: Said) -> Result { ))) if id == qid => { return Err(Error::Kademlia(format!("{e:?}"))); } - Ok(_) => {} - Err(_) => { - return Err(Error::Kademlia("provider lookup timed out".to_string())); - } + _ => {} } } } diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/tests/gossip_validation.rs b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/tests/gossip_validation.rs new file mode 100644 index 000000000..826496741 --- /dev/null +++ b/smart-byte/smart-byte-rs/crates/smart-byte-libp2p/tests/gossip_validation.rs @@ -0,0 +1,145 @@ +//! A forged envelope stops at the first honest node; a genuine one crosses it. +//! +//! A — B — C, with A and C not connected. B runs the crate's own event loop, +//! which settles every gossip message before gossipsub may forward it. A +//! publishes three messages: an envelope with its cargo changed under the +//! genuine id, the genuine envelope padded with an extra member, and the +//! genuine envelope. Only the last may reach C. + +use std::time::Duration; + +use chrono::TimeZone; +use libp2p::futures::StreamExt; +use libp2p::gossipsub::{self, IdentTopic}; +use libp2p::swarm::SwarmEvent; +use smart_byte_core::{Cargo, Envelope, JouleCost, OwnershipChain, Provenance, Said}; +use smart_byte_libp2p::behaviour::{SmartByteBehaviourEvent, default_gossipsub_config}; +use smart_byte_libp2p::transport::{settle_gossip, subscribe_topic}; +use smart_byte_libp2p::{Libp2pNode, NodeConfig, NodeEvent, run}; +use tokio::time::Instant; + +async fn node() -> Libp2pNode { + Libp2pNode::new(NodeConfig { + listen_addrs: vec!["/ip4/127.0.0.1/tcp/0".parse().unwrap()], + bootstrap_peers: vec![], + identity: libp2p::identity::Keypair::generate_ed25519(), + enable_mdns: false, + enable_relay: false, + enable_dcutr: false, + gossipsub_config: default_gossipsub_config(), + }) + .await + .expect("spawn node") +} + +/// Poll `node` once, settling any gossip message; return its data if it was one. +fn received(node: &mut Libp2pNode, ev: SwarmEvent) -> Option> { + if let SwarmEvent::Behaviour(SmartByteBehaviourEvent::Gossipsub(gossipsub::Event::Message { + propagation_source, + message_id, + message, + })) = ev + { + settle_gossip( + &mut node.swarm, + &message_id, + &propagation_source, + &message.data, + ); + return Some(message.data); + } + None +} + +#[tokio::test(flavor = "multi_thread")] +async fn a_forged_envelope_is_not_relayed() { + let (mut a, mut b, mut c) = (node().await, node().await, node().await); + let b_addr = b.listen_addrs[0].clone(); + a.swarm.dial(b_addr.clone()).expect("dial a -> b"); + c.swarm.dial(b_addr).expect("dial c -> b"); + let topic = "smart-byte/test/relay"; + for n in [&mut a, &mut b, &mut c] { + subscribe_topic(n, topic).expect("subscribe"); + } + + let (tx, mut rx) = tokio::sync::mpsc::channel::(1024); + tokio::spawn(async move { + let mut b = b; + let _ = run(&mut b, tx).await; + }); + tokio::spawn(async move { while rx.recv().await.is_some() {} }); + + // Let the connections settle and the topic mesh form. + let warm = Instant::now() + Duration::from_secs(4); + while Instant::now() < warm { + tokio::select! { + _ = a.swarm.select_next_some() => {} + _ = c.swarm.select_next_some() => {} + _ = tokio::time::sleep(Duration::from_millis(20)) => {} + } + } + assert!( + !a.swarm.is_connected(&c.local_peer_id), + "A and C must not be directly connected" + ); + + let issued = chrono::Utc.with_ymd_and_hms(2026, 9, 30, 0, 0, 0).unwrap(); + let genuine = Envelope::new( + Provenance::new(Said::hash(b"issuer"), issued, vec![]), + OwnershipChain::empty(), + Cargo::Usd { minor: 100 }, + JouleCost::measured(1), + ) + .unwrap(); + let mut forged = genuine.clone(); + forged.cargo = Cargo::Usd { minor: 1_000_000 }; + let forged = forged.to_cbor().unwrap(); + let genuine_bytes = genuine.to_cbor().unwrap(); + let mut padded = genuine_bytes.clone(); + padded[0] = 0xa6; + padded.extend_from_slice(&[0x61, b'x', 0x00]); + + let gossip = &mut a.swarm.behaviour_mut().gossipsub; + for bytes in [&forged, &padded, &genuine_bytes] { + gossip + .publish(IdentTopic::new(topic), bytes.clone()) + .expect("publish"); + } + + // Wait for the genuine envelope, then a grace period for anything that + // would have been relayed alongside it. + let mut seen: Vec> = Vec::new(); + let deadline = Instant::now() + Duration::from_secs(15); + let mut grace: Option = None; + loop { + let now = Instant::now(); + if now > deadline || grace.is_some_and(|g| now > g) { + break; + } + tokio::select! { + _ = a.swarm.select_next_some() => {} + ev = c.swarm.select_next_some() => { + if let Some(data) = received(&mut c, ev) { + if data == genuine_bytes && grace.is_none() { + grace = Some(Instant::now() + Duration::from_millis(1500)); + } + seen.push(data); + } + } + _ = tokio::time::sleep(Duration::from_millis(20)) => {} + } + } + + assert!( + seen.contains(&genuine_bytes), + "the genuine envelope never reached C" + ); + assert!( + !seen.contains(&forged), + "B relayed an envelope whose content does not hash to its id" + ); + assert!( + !seen.contains(&padded), + "B relayed a padded copy of a genuine envelope" + ); +} -- 2.54.0 (Apple Git-157) From b7df79083ad0b3caf90d529f64be5075644ed636 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Wed, 30 Sep 2026 21:33:24 -0400 Subject: [PATCH 11/12] smart-byte-keri-witness: reject `rec` by default, as spec v1 requires MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Identity spec §3.2 reserves `rec`, and a v1 verifier MUST reject it as malformed. `LogVerifier::new()` accepted it, because the strict flag defaulted to false. Strict is now the default. Accepting recoveries is an explicit opt-in (`strict(false)`), documented as an extension outside v1; the recovery test already opts in. Co-Authored-By: Claude Opus 5.5 --- .../smart-byte-keri-witness/src/verifier.rs | 24 ++++++++++++++----- 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-keri-witness/src/verifier.rs b/smart-byte/smart-byte-rs/crates/smart-byte-keri-witness/src/verifier.rs index d40549d85..25c5784b4 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-keri-witness/src/verifier.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-keri-witness/src/verifier.rs @@ -43,25 +43,37 @@ pub struct VerificationReport { } /// Replay verifier. -#[derive(Default)] pub struct LogVerifier { - /// When `true`, reject `rec` events outright per spec §3.2. - /// When `false`, accept recoveries (this crate's extension). + /// `true` (the default): reject `rec` events outright. Identity spec §3.2 + /// reserves `rec`, and a v1 verifier MUST reject it as malformed. `false` + /// accepts recoveries verified against the keys they reveal — an + /// extension outside spec v1, for exercising recovery before the spec + /// defines it. pub strict: bool, /// When `true`, also require the witness-receipt threshold be met /// for every event in the log. pub require_witness_quorum: bool, } +impl Default for LogVerifier { + fn default() -> Self { + Self { + strict: true, + require_witness_quorum: false, + } + } +} + impl LogVerifier { - /// Build a permissive verifier that accepts recoveries and does - /// not require witness quorum. + /// Build a spec-v1 verifier: rejects `rec`, and does not require witness + /// quorum. #[must_use] pub fn new() -> Self { Self::default() } - /// Enable strict spec mode (rejects `rec`). + /// `true` (the default) rejects `rec` per spec v1; `false` opts into the + /// recovery extension. #[must_use] pub fn strict(mut self, on: bool) -> Self { self.strict = on; -- 2.54.0 (Apple Git-157) From 7dba0f3db9f26c7f5d81aaaaa9d3e35d05368c2a Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Wed, 30 Sep 2026 21:34:19 -0400 Subject: [PATCH 12/12] smart-byte: the conformance pack, corrected against an independent review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The review found the pack and its checkers wrong in ways two agreeing implementations could not catch. Spec: - Identity spec §2.1, §2.3, §2.4 and §4 step 1 told a verifier to re-encode an object canonically. A verifier written to them accepts said-009, the re-encoding attack §2.5 exists to refuse. All four now require received bytes to be canonical already. - New §2.6 fixes how a SAID-addressed object is received: canonical bytes, then the SAID they state, then the SAID they arrived under, and only then deduplication. - §2.3 forbids simple values other than false, true and null. - §3.3/§3.4 say a key is a byte string, as said-007 already encoded it. Vectors: - Lockstep vectors separate the hash a node broadcast from the one it computed, and give each node's outcome and dissent log from its own view. The old single view called a node a "liar" while modelling divergence; lockstep-006/007 now pin treatise §13.4's transport liar, who commits in its own view while its peers log it. - Gossip vectors carry generic SAID-addressed objects. Envelope-shaped ones with null or filler provenance are bytes a provenance-checking node must refuse (treatise §8.3), so they could not be MUST-pass. `expected_held` now carries the bytes held. - New canon-020 (simple value) and sig-007 (a key spelled with y = p + 3, a second spelling of a valid point that ed25519-dalek alone accepts). - SAID vector descriptions no longer claim fidelity to treatise §8 or a frame the data never mentions. Checkers: - Both now check the pack whole. An empty pack, a deleted vector the spec cites, a deleted class, or an unlisted class all fail; before, an empty pack passed both checkers. - The Python checker no longer uses `assert`. Under `python -O` it passed a corrupted pack. - Both refuse JSON that names a member twice. - The Rust verifier no longer panics on an empty stated SAID. - The gossip expectations come from smart-byte-core's inbox, so every value in the pack is now the reference implementation's. Falsified: - The Rust verifier catches 19 of 19 planted defects, each on its own vector, plus the empty-pack, deleted-vector, deleted-class, unlisted-class and duplicate-member cases. - The Python checker catches 15 of 15 planted defects in both normal and -O modes, plus the same pack-level cases. regeneration.md is consolidated into one accurate entry, which corrects the earlier claim that no draft vector was passable. VERSIONS.md rule 4 and the README now scope supersession to frozen packs. Co-Authored-By: Claude Opus 5.5 --- smart-byte/conformance/README.md | 71 ++-- smart-byte/conformance/VERSIONS.md | 14 +- .../conformance/crosscheck/verify_v1.py | 298 +++++++++----- .../v1/canonicalisation_vectors.json | 8 +- smart-byte/conformance/v1/gossip_vectors.json | 85 ++-- .../conformance/v1/lockstep_vectors.json | 276 +++++++++++-- smart-byte/conformance/v1/regeneration.md | 171 ++++---- smart-byte/conformance/v1/said_vectors.json | 12 +- .../conformance/v1/signature_vectors.json | 19 +- smart-byte/smart-byte-rs/Cargo.lock | 1 + .../crates/smart-byte-conformance/Cargo.toml | 1 + .../smart-byte-conformance/src/gossip.rs | 71 ---- .../crates/smart-byte-conformance/src/json.rs | 120 +++++- .../crates/smart-byte-conformance/src/lib.rs | 1 - .../crates/smart-byte-conformance/src/pack.rs | 372 ++++++++++-------- .../smart-byte-conformance/src/verify.rs | 302 ++++++++++---- .../smart-byte-conformance/tests/pack.rs | 185 ++++++--- smart-byte/spec/identity_and_key_rotation.md | 31 +- 18 files changed, 1381 insertions(+), 657 deletions(-) delete mode 100644 smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/gossip.rs diff --git a/smart-byte/conformance/README.md b/smart-byte/conformance/README.md index 8f9f5c438..9c19c8cb2 100644 --- a/smart-byte/conformance/README.md +++ b/smart-byte/conformance/README.md @@ -29,7 +29,7 @@ A pack for a given version contains five JSON files plus `regeneration.md`: | `said_vectors.json` | SAID derivation and verification (identity §2.1–§2.5; treatise §8.2) | | `signature_vectors.json` | Ed25519 signatures over SAIDs (treatise §8.3, §8.4; RFC 8032) | | `lockstep_vectors.json` | Supermajority commit, frame outcomes, dissent log (treatise §13, §14) | -| `gossip_vectors.json` | Content-addressed intake of gossiped objects (treatise §10.2, §11.4, §21.4) | +| `gossip_vectors.json` | Receiving SAID-addressed objects (identity §2.6; treatise §11.4, §21.4) | Every value in the pack is generated by the Rust reference implementation (`../smart-byte-rs`, crate `smart-byte-conformance`) and reproduced by an @@ -67,10 +67,12 @@ A vector pack is a hard gate. JSON has no byte-string type, so an `input` writes a CBOR byte string as an object whose only member is `$bytes`, holding lowercase hex: -`{"$bytes": "00ff"}`. Every other JSON value maps to CBOR directly: object → -map with text keys, array → array, string → text string, integer → integer, +`{"$bytes": "00ff"}`. `$bytes` is reserved: no map in the pack has a member +of that name. Every other JSON value maps to CBOR directly: object → map with +text keys, array → array, string → text string, integer → integer, `true` / `false` / `null` → the simple values. The pack contains no JSON number -with a fraction or exponent, because floats have no canonical form. +with a fraction or exponent, because identity spec §2.3 forbids floats, and no +object names a member twice. Where a vector supplies raw bytes instead (`*_cbor_hex`, `input_cbor_hex`), they are lowercase hex. @@ -90,22 +92,30 @@ they are lowercase hex. `input_cbor_hex` must fail SAID verification with `error`. The adversarial vectors also give, for illustration, the SAIDs a lenient verifier would compute (`said_of_received_bytes`, `said_after_reencoding`). -- **`signature_vectors.json`** — `signed_message_hex` is the message: the 32 - digest bytes `envelope_said` spells, not its text. `verifying_key` is the - key `signing_key_seed` derives. Verify `signature` under `verifying_key`: - the result must be `expected_verify`, failing with `error` when false. -- **`lockstep_vectors.json`** — `node_hashes` are the opaque 32-byte state - hashes each node reported, compared only for equality, and `node_count` is - the cluster size n. The IUT computes `expected_threshold` (floor(2n/3)+1), - `expected_commit`, `expected_state_hash`, each node's `expected_outcomes` +- **`signature_vectors.json`** — the Ed25519 rules every Smart Byte signature + follows: RFC 8032, with strict decoding of keys and scalars. The message is + `signed_message_hex`, the 32 digest bytes `envelope_said` spells — not its + text. `verifying_key` is the key handed to the verifier; when + `signing_key_seed` is not null, it is the key that seed derives. Verify + `signature` under `verifying_key`: the result must be `expected_verify`, + failing with `error` when false. A positive vector's `signature` is the one + deterministic signature its seed makes. +- **`lockstep_vectors.json`** — `node_count` is the cluster size n. + `node_reported_hashes` are the opaque 32-byte state hashes each node + broadcast, and `node_computed_hashes` the ones each computed; they differ + only for a node that lies on the wire (treatise §13.4). Hashes are compared + only for equality. The IUT computes `expected_threshold` (floor(2n/3)+1), + then `expected_commit` and `expected_state_hash` from the reported hashes — + the frame's public record. It also computes each node's `expected_outcomes` (`clean_commit`, `commit_with_dissent`, `minority_divergence` or - `cluster_split`) and the dissent-log entries the committing nodes write - (`expected_dissent`). + `cluster_split`) and `expected_dissent_logs` from that node's own view: its + peers' reported hashes plus the hash it computed. - **`gossip_vectors.json`** — `deliveries` arrive at `receiver` in order, each with the SAID it was announced or requested under (`claimed_said`) and its bytes (`cbor_hex`). Each result is `accepted` or `duplicate` with the SAID, - or `rejected` with an error class. `expected_held` lists the SAIDs the - receiver holds afterwards, in the order it accepted them. + or `rejected` with an error class, per identity spec §2.6. `expected_held` + lists what the receiver holds afterwards, in acceptance order: each SAID + with exactly the bytes delivered. ### Error classes @@ -116,6 +126,7 @@ they are lowercase hex. | `unsorted_map_keys` | map keys not in the bytewise order of their encodings | | `duplicate_map_key` | a key appears twice in one map | | `float_forbidden` | a floating-point value | +| `simple_value_forbidden` | a simple value other than `false`, `true` and `null` | | `tag_forbidden` | a CBOR tag | | `indefinite_length` | an indefinite-length item | | `invalid_utf8` | a text string that is not UTF-8 | @@ -123,10 +134,20 @@ they are lowercase hex. | `said_mismatch` | content that does not hash to the SAID it states or arrived under | | `signature_invalid` | a signature that does not verify, including a non-canonical S | | `malformed_signature` | a signature that is not 64 bytes | +| `malformed_verifying_key` | a key that is not 32 bytes, or encodes y outside the field | The reference implementation also reports `truncated`, `malformed`, -`too_deep`, `not_a_map`, `missing_said_field`, `malformed_said` and -`malformed_verifying_key` for inputs no vector exercises. +`too_deep`, `not_a_map`, `missing_said_field` and `malformed_said` for +inputs no vector exercises. An input with several defects reports the first +in stream order. + +### Completeness + +A pack is checked whole, not only vector by vector. Every file holds at least +one vector, every vector the identity spec cites is present, and every error +class in the table above is exercised by some vector — and no vector names a +class the table does not list. Both checkers enforce this, so a pack cannot +pass by losing vectors. ## Versioning rule @@ -139,11 +160,13 @@ calendar release of the conformance pack. - `v2/` will appear when (and only when) the spec moves to a new version that introduces an incompatible wire-format change. The compatibility matrix lives in `VERSIONS.md`. -- A bug in a vector pack (e.g. an `expected_*` field that disagrees with - the spec) is fixed by superseding the vector with a new one whose `name` - carries a suffix (`-fixed`, `-corrected`); the original is retained with - a `"superseded_by"` field so historical IUTs that pinned to the old - vector can detect the supersession. +- In a frozen version, a bug in a vector pack (e.g. an `expected_*` field + that disagrees with the spec) is fixed by superseding the vector with a + new one whose `name` carries a suffix (`-fixed`, `-corrected`); the + original is retained with a `"superseded_by"` field so historical IUTs + that pinned to the old vector can detect the supersession. A DRAFT pack is + regenerated in place instead, and every change is recorded in its + `regeneration.md`. ## CI gate @@ -153,7 +176,7 @@ pass the gate. The gate validates: - Every JSON file in `v*/` parses as valid JSON. - The second implementation, `crosscheck/verify_v1.py`, agrees with every - vector. + vector, and finds the pack complete (see Completeness). - Every spec change is accompanied by a vector-pack update (a PR that modifies `smart-byte/spec/` without modifying `smart-byte/conformance/` is rejected at the gate). diff --git a/smart-byte/conformance/VERSIONS.md b/smart-byte/conformance/VERSIONS.md index d323ddc2e..ef611f852 100644 --- a/smart-byte/conformance/VERSIONS.md +++ b/smart-byte/conformance/VERSIONS.md @@ -7,7 +7,7 @@ Smart Byte spec versions. | Vector pack | Spec version | Spec source | Status | Notes | | ----------- | ------------ | ------------------------------------------------------------- | -------- | --------------------------------------------------------------------- | -| `v1/` | v1.0-draft | `../spec/treatise_v1_parts_I_II_III_combined.md` (Parts I-III), `../spec/identity_and_key_rotation.md` | DRAFT | Generated by the reference implementation (`smart-byte-conformance`) and reproduced by `crosscheck/verify_v1.py`; regenerated in place on 2026-09-30, see `v1/regeneration.md`. Supersession (rule 4) applies once v1 freezes. | +| `v1/` | v1.0-draft | `../spec/treatise_v1_parts_I_II_III_combined.md` (Parts I-III), `../spec/identity_and_key_rotation.md` | DRAFT | Generated by the reference implementation (`smart-byte-conformance`) and reproduced by `crosscheck/verify_v1.py`. Regenerated in place while DRAFT (rule 4); every change is in `v1/regeneration.md`. | Future entries are appended below this table when a new spec version is introduced. Older entries are never removed; an implementation that pins to @@ -28,11 +28,13 @@ indefinitely. *additional* corners of the *same* canonical wire format, but no existing vector's `expected_*` outputs may be changed. -4. **Bug fixes via supersession.** A vector with an `expected_*` field that - disagrees with the spec is fixed by adding a new vector (with a - distinguishing `name` suffix) and adding a `"superseded_by": ""` - field to the original. Implementations pin to vector names, not array - positions, so supersession is non-breaking. +4. **Bug fixes via supersession, once frozen.** In a frozen version, a + vector with an `expected_*` field that disagrees with the spec is fixed by + adding a new vector (with a distinguishing `name` suffix) and adding a + `"superseded_by": ""` field to the original. Implementations pin + to vector names, not array positions, so supersession is non-breaking. A + DRAFT pack is regenerated in place, with every change recorded in its + `regeneration.md`. 5. **Spec-version detection at the wire.** The wire format itself carries version information (see spec §8.3, §8.4 — the one-byte algorithm diff --git a/smart-byte/conformance/crosscheck/verify_v1.py b/smart-byte/conformance/crosscheck/verify_v1.py index ac7c68747..9f8170e29 100644 --- a/smart-byte/conformance/crosscheck/verify_v1.py +++ b/smart-byte/conformance/crosscheck/verify_v1.py @@ -22,6 +22,7 @@ import base64 import hashlib import json import pathlib +import re import sys # --------------------------------------------------------------------------- @@ -299,7 +300,9 @@ def decode(data, strict=True): raise Refused("float_forbidden") if ai in (28, 29, 30, 31): raise Refused("malformed") - raise Refused("unsupported_simple_value") + if ai == 24 and need(1)[0] < 32: + raise Refused("malformed") # RFC 8949 §3.3: not well-formed + raise Refused("simple_value_forbidden") if major == 6: raise Refused("tag_forbidden") if ai in (28, 29, 30): @@ -348,10 +351,13 @@ def decode(data, strict=True): def from_json(j): - """The pack's typed JSON: {"$bytes": hex} is a byte string.""" + """The pack's typed JSON: {"$bytes": hex} is a byte string; `$bytes` is + reserved and never a map member.""" if isinstance(j, dict): - if list(j) == ["$bytes"] and isinstance(j["$bytes"], str): + if "$bytes" in j: h = j["$bytes"] + if len(j) != 1 or not isinstance(h, str): + raise ValueError("`$bytes` is reserved: a lone member holding hex") if len(h) % 2 or any(c not in "0123456789abcdef" for c in h): raise ValueError(f"$bytes must be lowercase hex: {h!r}") return bytes.fromhex(h) @@ -359,12 +365,24 @@ def from_json(j): if isinstance(j, list): return [from_json(x) for x in j] if isinstance(j, float): - raise ValueError("floats have no canonical form") + raise ValueError("floats are forbidden") return j +def load_json(path): + """Parse JSON, refusing any object that names a member twice.""" + def no_duplicates(pairs): + out = {} + for k, v in pairs: + if k in out: + raise ValueError(f"duplicate member {k!r}") + out[k] = v + return out + return json.loads(path.read_text(encoding="utf-8"), object_pairs_hook=no_duplicates) + + # --------------------------------------------------------------------------- -# SAIDs (identity spec §2.1, §2.2) +# SAIDs (identity spec §2.1, §2.2) and receiving them (§2.6) # --------------------------------------------------------------------------- PLACEHOLDER = "#" * 53 @@ -402,10 +420,10 @@ def verify_said(data, field): obj = decode(data) if not isinstance(obj, Map): raise Refused("not_a_map") - stated = dict((k, v) for k, v in obj if isinstance(k, str)).get(field, Refused) - if stated is Refused: + stated = [v for k, v in obj if k == field] + if not stated: raise Refused("missing_said_field") - digest = parse_said(stated) + digest = parse_said(stated[0]) if digest is None: raise Refused("malformed_said") if derive(obj, field)[1] != digest: @@ -413,10 +431,39 @@ def verify_said(data, field): return digest +def receive(held, claimed, data): + """Identity spec §2.6, in order: canonical bytes, SAID match, the claimed + SAID, and only then deduplication. Returns the result object.""" + try: + digest = verify_said(data, "d") + claim = parse_said(claimed) + if claim is None: + raise Refused("malformed_said") + if claim != digest: + raise Refused("said_mismatch") + except Refused as e: + return {"outcome": "rejected", "error": e.cls} + text = said_text(digest) + if any(h["said"] == text for h in held): + return {"outcome": "duplicate", "said": text} + held.append({"said": text, "cbor_hex": data.hex()}) + return {"outcome": "accepted", "said": text} + + # --------------------------------------------------------------------------- -# The checks +# The checks. Each raises Mismatch; none uses `assert`, so `python -O` +# cannot switch them off. # --------------------------------------------------------------------------- +class Mismatch(Exception): + pass + + +def check(condition, message): + if not condition: + raise Mismatch(message) + + def refused_class(fn, *args): try: fn(*args) @@ -428,125 +475,134 @@ def refused_class(fn, *args): def check_canonicalisation(v): if "input" in v: b = encode(from_json(v["input"])) - assert b.hex() == v["expected_cbor_hex"], f"computed {b.hex()}" - assert encode(decode(b)) == b, "decode then encode changed the bytes" + check(b.hex() == v["expected_cbor_hex"], f"expected_cbor_hex: computed {b.hex()}") + check(encode(decode(b)) == b, "decode then encode changed the bytes") else: got = refused_class(decode, bytes.fromhex(v["input_cbor_hex"])) - assert got == v["error"], f"refused with {got}" + check(got == v["error"], f"refused with {got}, vector says {v['error']}") def check_said(v): field = v["said_field"] if "input" in v: obj = from_json(v["input"]) - assert dict(obj).get(field) == PLACEHOLDER, "input must hold the placeholder" + check([x for k, x in obj if k == field] == [PLACEHOLDER], "input must hold the placeholder") hashed, digest, stamped = derive(obj, field) - assert hashed.hex() == v["canonical_cbor_hex"], "canonical_cbor_hex" - assert digest.hex() == v["blake3_hex"], "blake3_hex" - assert said_text(digest) == v["expected_said"], f"expected_said: computed {said_text(digest)}" - assert stamped.hex() == v["said_cbor_hex"], "said_cbor_hex" - assert verify_said(stamped, field) == digest, "said_cbor_hex does not verify" + check(hashed.hex() == v["canonical_cbor_hex"], "canonical_cbor_hex") + check(digest.hex() == v["blake3_hex"], "blake3_hex") + check(said_text(digest) == v["expected_said"], f"expected_said: computed {said_text(digest)}") + check(stamped.hex() == v["said_cbor_hex"], "said_cbor_hex") + check(verify_said(stamped, field) == digest, "said_cbor_hex does not verify") else: data = bytes.fromhex(v["input_cbor_hex"]) got = refused_class(verify_said, data, field) - assert got == v["error"], f"refused with {got}" - lenient = decode(data, strict=False) - stated = dict(lenient)[field] - if "said_of_received_bytes" in v: - at = data.index(stated.encode()) - raw = data[:at] + PLACEHOLDER.encode() + data[at + 53:] - assert said_text(blake3(raw)) == v["said_of_received_bytes"], "said_of_received_bytes" - if "said_after_reencoding" in v: - assert said_text(derive(lenient, field)[1]) == v["said_after_reencoding"], "said_after_reencoding" + check(got == v["error"], f"refused with {got}, vector says {v['error']}") + if "said_of_received_bytes" in v or "said_after_reencoding" in v: + lenient = decode(data, strict=False) + stated = [x for k, x in lenient if k == field] + check(len(stated) == 1 and isinstance(stated[0], str) and len(stated[0]) == 53, + "no 53-character SAID to substitute") + if "said_of_received_bytes" in v: + at = data.find(stated[0].encode()) + check(at >= 0, "the stated SAID is not in the bytes") + raw = data[:at] + PLACEHOLDER.encode() + data[at + 53:] + check(said_text(blake3(raw)) == v["said_of_received_bytes"], "said_of_received_bytes") + if "said_after_reencoding" in v: + check(said_text(derive(lenient, field)[1]) == v["said_after_reencoding"], + "said_after_reencoding") def check_signature(v, saids): - assert v["envelope_said"] in saids, "envelope_said is not a SAID vector's expected_said" + check(v["envelope_said"] in saids, "envelope_said is not a SAID vector's expected_said") digest = parse_said(v["envelope_said"]) - assert digest is not None and digest.hex() == v["signed_message_hex"], "signed_message_hex" - seed = bytes.fromhex(v["signing_key_seed"]) - assert ed25519_public(seed).hex() == v["verifying_key"], "verifying_key" + check(digest is not None and digest.hex() == v["signed_message_hex"], "signed_message_hex") + seed = v["signing_key_seed"] + if seed is not None: + check(ed25519_public(bytes.fromhex(seed)).hex() == v["verifying_key"], + "verifying_key is not the key the seed derives") sig = bytes.fromhex(v["signature"]) got = ed25519_verify(bytes.fromhex(v["verifying_key"]), digest, sig) - if v["expected_verify"]: - assert got is None and v["error"] is None, f"a positive vector fails: {got}" - assert ed25519_sign(seed, digest) == sig, "the deterministic signature differs" + if v["expected_verify"] is True: + check(got is None and v["error"] is None, f"a positive vector fails: {got}") + check(seed is not None, "a positive vector must carry its seed") + check(ed25519_sign(bytes.fromhex(seed), digest) == sig, "the deterministic signature differs") else: - assert got == v["error"], f"refused with {got}" + check(v["expected_verify"] is False, "expected_verify is not a boolean") + check(got == v["error"], f"refused with {got}, vector says {v['error']}") -def check_lockstep(v): - n, frame = v["node_count"], v["frame_number"] - reports = v["node_hashes"] - assert 0 < len(reports) <= n +def tally(n, reports): + """(supermajority hash or None, sorted dissenters) — treatise §13.1.""" threshold = 2 * n // 3 + 1 - assert v["expected_threshold"] == threshold, "expected_threshold" votes = {} for h in reports.values(): votes[h] = votes.get(h, 0) + 1 winners = [h for h, c in votes.items() if c >= threshold] winner = winners[0] if winners else None - assert v["expected_commit"] == (winner is not None), "expected_commit" - assert v["expected_state_hash"] == winner, "expected_state_hash" dissenters = sorted(node for node, h in reports.items() if winner and h != winner) - outcomes = {} - for node, h in reports.items(): - if winner is None: + return winner, dissenters + + +def check_lockstep(v): + n, frame = v["node_count"], v["frame_number"] + reported, computed = v["node_reported_hashes"], v["node_computed_hashes"] + check(sorted(reported) == sorted(computed), "reported and computed name different nodes") + check(0 < len(reported) <= n, "more reports than nodes") + check(v["expected_threshold"] == 2 * n // 3 + 1, "expected_threshold") + winner, _ = tally(n, reported) + check(v["expected_commit"] == (winner is not None), "expected_commit") + check(v["expected_state_hash"] == winner, "expected_state_hash") + outcomes, logs = {}, {} + for node, own in computed.items(): + # A node's view: its peers' reports and the hash it computed (§13.4). + view = dict(reported) + view[node] = own + best, dissent = tally(n, view) + if best is None: outcomes[node] = "cluster_split" - elif h != winner: + elif own != best: outcomes[node] = "minority_divergence" else: - outcomes[node] = "commit_with_dissent" if dissenters else "clean_commit" - assert v["expected_outcomes"] == outcomes, f"expected_outcomes: computed {outcomes}" - entries = [] - if dissenters: - entries.append({"frame": frame, "dissenting_peers": dissenters, - "local_hash": winner, "supermajority_hash": winner}) - assert v["expected_dissent"] == entries, f"expected_dissent: computed {entries}" + outcomes[node] = "commit_with_dissent" if dissent else "clean_commit" + logs[node] = [{"frame": frame, "dissenting_peers": dissent, + "local_hash": own, "supermajority_hash": best}] \ + if outcomes[node] == "commit_with_dissent" else [] + check(v["expected_outcomes"] == outcomes, f"expected_outcomes: computed {outcomes}") + check(v["expected_dissent_logs"] == logs, f"expected_dissent_logs: computed {logs}") def check_gossip(v): + check(len(v["deliveries"]) == len(v["expected_results"]), "one result per delivery") held = [] - assert len(v["deliveries"]) == len(v["expected_results"]) for d, want in zip(v["deliveries"], v["expected_results"]): - data = bytes.fromhex(d["cbor_hex"]) - try: - digest = verify_said(data, "d") - claimed = parse_said(d["claimed_said"]) - if claimed is None: - raise Refused("malformed_said") - if claimed != digest: - raise Refused("said_mismatch") - text = said_text(digest) - got = {"outcome": "duplicate" if text in held else "accepted", "said": text} - if text not in held: - held.append(text) - except Refused as e: - got = {"outcome": "rejected", "error": e.cls} - assert got == want, f"published {want}, computed {got}" - assert v["expected_held"] == held, f"expected_held: computed {held}" + got = receive(held, d["claimed_said"], bytes.fromhex(d["cbor_hex"])) + check(got == want, f"published {want}, computed {got}") + check(v["expected_held"] == held, f"expected_held: computed {held}") def self_test(): # BLAKE3 known answers: the empty input, and the official test-vector input # pattern (byte i is i % 251) at lengths that cross block and chunk edges. - assert blake3(b"").hex() == "af1349b9f5f9a1a6a0404dea36dcc9499bcb25c9adc112b7cc9a93cae41f3262" + check(blake3(b"").hex() == "af1349b9f5f9a1a6a0404dea36dcc9499bcb25c9adc112b7cc9a93cae41f3262", + "BLAKE3 of the empty input") for n, want in BLAKE3_KAT: got = blake3(bytes(i % 251 for i in range(n))).hex() - assert got == want, f"BLAKE3 known answer for length {n}: {got}" + check(got == want, f"BLAKE3 known answer for length {n}: {got}") # Ed25519 known answers: RFC 8032 §7.1 tests 1 and 2. s1 = bytes.fromhex("9d61b19deffd5a60ba844af492ec2cc44449c5697b326919703bac031cae7f60") - assert ed25519_public(s1).hex() == "d75a980182b10ab7d54bfed3c964073a0ee172f3daa62325af021a68f707511a" - assert ed25519_sign(s1, b"").hex() == ( + check(ed25519_public(s1).hex() == "d75a980182b10ab7d54bfed3c964073a0ee172f3daa62325af021a68f707511a", + "RFC 8032 test 1 key") + check(ed25519_sign(s1, b"").hex() == ( "e5564300c360ac729086e2cc806e828a84877f1eb8e5d974d873e065224901555" - "fb8821590a33bacc61e39701cf9b46bd25bf5f0595bbe24655141438e7a100b") + "fb8821590a33bacc61e39701cf9b46bd25bf5f0595bbe24655141438e7a100b"), "RFC 8032 test 1 signature") s2 = bytes.fromhex("4ccd089b28ff96da9db6c346ec114e0f5b8a319f35aba624da8cf6ed4fb8a6fb") sig2 = ed25519_sign(s2, b"\x72") - assert sig2.hex() == ( + check(sig2.hex() == ( "92a009a9f0d4cab8720e820b5f642540a2b27b5416503f8fb3762223ebdb69da" - "085ac1e43e15996e458f3613d0f11d8c387b2eaeb4302aeeb00d291612bb0c00") - assert ed25519_verify(ed25519_public(s2), b"\x72", sig2) is None - assert ed25519_verify(ed25519_public(s2), b"\x73", sig2) == "signature_invalid" + "085ac1e43e15996e458f3613d0f11d8c387b2eaeb4302aeeb00d291612bb0c00"), "RFC 8032 test 2 signature") + check(ed25519_verify(ed25519_public(s2), b"\x72", sig2) is None, "RFC 8032 test 2 verifies") + check(ed25519_verify(ed25519_public(s2), b"\x73", sig2) == "signature_invalid", + "a changed message fails") # BLAKE3 hashes of the official test-vector input pattern (byte i is i % 251), @@ -575,22 +631,81 @@ BLAKE3_KAT = [ CHECKS = { "canonicalisation_vectors.json": check_canonicalisation, "said_vectors.json": check_said, + "signature_vectors.json": None, # needs the SAID set; dispatched in main "lockstep_vectors.json": check_lockstep, "gossip_vectors.json": check_gossip, } +HERE = pathlib.Path(__file__).resolve().parent + + +def spec_cited_said_vectors(): + """The SAID vector prefixes the identity spec cites, ranges expanded.""" + spec = (HERE.parent.parent / "spec" / "identity_and_key_rotation.md").read_text(encoding="utf-8") + words = [w for w in re.split(r"[^A-Za-z0-9-]+", spec) if w] + cited = set() + for i, w in enumerate(words): + m = re.fullmatch(r"said-(\d+)", w) + if not m: + continue + cited.add(f"said-{int(m.group(1)):03d}") + end = re.fullmatch(r"said-(\d+)", words[i + 2]) if i + 2 < len(words) and words[i + 1] == "to" else None + if end: + cited.update(f"said-{k:03d}" for k in range(int(m.group(1)), int(end.group(1)) + 1)) + return cited + + +def readme_error_classes(): + readme = (HERE.parent / "README.md").read_text(encoding="utf-8") + section = readme.split("### Error classes", 1)[1] + rows = [] + for line in section.splitlines(): + if line.startswith("|"): + rows.append(line) + elif rows: + break + return {line.split("`")[1] for line in rows if line.count("`") >= 2} + + +def check_complete(files): + """Every file non-empty; every vector the spec cites present; every README + error class exercised, and no other class used.""" + for name, vectors in files.items(): + check(len(vectors) > 0, f"{name} holds no vectors") + names = [v["name"] for v in files["said_vectors.json"]] + missing = sorted(p for p in spec_cited_said_vectors() + if not any(n.startswith(p + "-") for n in names)) + check(not missing, f"the identity spec cites vectors the pack lacks: {missing}") + used = set() + for vectors in files.values(): + for v in vectors: + if isinstance(v.get("error"), str): + used.add(v["error"]) + for r in v.get("expected_results", []): + if "error" in r: + used.add(r["error"]) + listed = readme_error_classes() + check(listed and used == listed, + f"README classes unexercised: {sorted(listed - used)}; unlisted classes used: {sorted(used - listed)}") + def main(pack): self_test() - files = {f: json.loads((pack / f).read_text(encoding="utf-8")) - for f in list(CHECKS) + ["signature_vectors.json"]} - saids = {v["expected_said"] for v in files["said_vectors.json"] if "expected_said" in v} failed = total = 0 - for name, vectors in files.items(): - names = [v["name"] for v in vectors] - if len(set(names)) != len(names): - print(f"FAIL {name}: duplicate vector names") + files = {} + for name in CHECKS: + try: + data = load_json(pack / name) + check(isinstance(data, list), "the top-level value must be an array") + names = [v["name"] for v in data] + check(len(set(names)) == len(names), "duplicate vector names") + check(all(isinstance(v.get("description"), str) for v in data), "a vector lacks a description") + files[name] = data + except (Mismatch, ValueError, KeyError, OSError) as e: failed += 1 + print(f"FAIL {name}: {e}") + saids = {v["expected_said"] for v in files.get("said_vectors.json", []) if "expected_said" in v} + for name, vectors in files.items(): for v in vectors: total += 1 try: @@ -599,13 +714,20 @@ def main(pack): else: CHECKS[name](v) print(f"ok {name:<32} {v['name']}") - except (AssertionError, Refused, KeyError, ValueError) as e: + except (Mismatch, Refused, KeyError, ValueError, TypeError) as e: failed += 1 print(f"FAIL {name:<32} {v.get('name')}: {e!r}") + if len(files) == len(CHECKS): + try: + check_complete(files) + print("ok (pack) complete") + except Mismatch as e: + failed += 1 + print(f"FAIL (pack): {e}") print(f"{total} vectors, {failed} failed") return 1 if failed else 0 if __name__ == "__main__": - default = pathlib.Path(__file__).resolve().parent.parent / "v1" + default = HERE.parent / "v1" sys.exit(main(pathlib.Path(sys.argv[1]) if len(sys.argv) > 1 else default)) diff --git a/smart-byte/conformance/v1/canonicalisation_vectors.json b/smart-byte/conformance/v1/canonicalisation_vectors.json index e68908f2e..6609e79dc 100644 --- a/smart-byte/conformance/v1/canonicalisation_vectors.json +++ b/smart-byte/conformance/v1/canonicalisation_vectors.json @@ -35,7 +35,7 @@ }, { "name": "canon-004-floats-forbidden", - "description": "Floating-point values have no canonical form: identity spec §2.3 forbids them, and a value that would be a float elsewhere is carried as a fixed-point integer. The input is a map holding 1.5 as a half-precision float, and a decoder refuses it.", + "description": "Floats are forbidden (identity spec §2.3): a value that would be a float elsewhere is carried as a fixed-point integer pair. The input is a map holding 1.5 as a half-precision float, and a decoder refuses it.", "input_cbor_hex": "a16a65786163745f68616c66f93e00", "error": "float_forbidden" }, @@ -164,5 +164,11 @@ "description": "A text string whose bytes (0xc3 0x28) are not UTF-8. CBOR text strings are UTF-8 (RFC 8949 §3.1), so the input is refused.", "input_cbor_hex": "62c328", "error": "invalid_utf8" + }, + { + "name": "canon-020-simple-value-refused", + "description": "The simple value undefined (0xf7). Identity spec §2.3 admits false, true and null and no other simple value, so the data model has nothing to decode it to and the input is refused.", + "input_cbor_hex": "f7", + "error": "simple_value_forbidden" } ] diff --git a/smart-byte/conformance/v1/gossip_vectors.json b/smart-byte/conformance/v1/gossip_vectors.json index 5581f543a..acd6ed9a6 100644 --- a/smart-byte/conformance/v1/gossip_vectors.json +++ b/smart-byte/conformance/v1/gossip_vectors.json @@ -1,45 +1,48 @@ [ { "name": "gossip-001-cross-node-propagation", - "description": "An envelope published at node_A reaches node_B. node_B accepts it because its bytes are canonical and hash to the SAID they state, which is the SAID it was announced under, and it holds the bytes unchanged. Gossip carries the object itself, never a re-encoding of it.", + "description": "An object published at node_A reaches node_B. Its bytes are canonical, they hash to the SAID they state, and that is the SAID they were announced under, so node_B accepts it (identity spec §2.6) and holds exactly the bytes it received. Gossip carries the object itself, never a re-encoding of it.", "receiver": "node_B", "deliveries": [ { "from": "node_A", - "claimed_said": "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq", - "cbor_hex": "a5616478354574626c613563346e61787372657861666c787a72686d376672706b626463656769746b67686f6c663663636d74366977706f787165636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730c75657374696d617465645f6d6963726f6a6f756c65730f6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" + "claimed_said": "Eze2x3zkvaaxvxu6hb75kidinrkmtczqu63b6qnzbg3g75e6hnwra", + "cbor_hex": "a261647835457a653278337a6b7661617876787536686237356b6964696e726b6d74637a717536336236716e7a6267336737356536686e777261616b582101d75a980182b10ab7d54bfed3c964073a0ee172f3daa62325af021a68f707511a" } ], "expected_results": [ { "outcome": "accepted", - "said": "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq" + "said": "Eze2x3zkvaaxvxu6hb75kidinrkmtczqu63b6qnzbg3g75e6hnwra" } ], "expected_held": [ - "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq" + { + "said": "Eze2x3zkvaaxvxu6hb75kidinrkmtczqu63b6qnzbg3g75e6hnwra", + "cbor_hex": "a261647835457a653278337a6b7661617876787536686237356b6964696e726b6d74637a717536336236716e7a6267336737356536686e777261616b582101d75a980182b10ab7d54bfed3c964073a0ee172f3daa62325af021a68f707511a" + } ] }, { "name": "gossip-002-said-conflict-rejected", - "description": "node_C holds said-003's object when a second object arrives under the same SAID: said-003's with its reading changed and its SAID left as it was. Content addressing settles the conflict without a first-come rule: the newcomer does not hash to the SAID it states, so it is refused. Two different canonical objects cannot share a SAID short of a BLAKE3 collision.", + "description": "node_C holds said-008's object when a second object arrives under the same SAID: said-008's with n changed and its SAID left as it was (said-011's bytes). Verification comes before deduplication (identity spec §2.6), so the newcomer is reported as a SAID mismatch, which makes the substitution attempt visible, instead of being absorbed as a duplicate of what node_C holds.", "receiver": "node_C", "deliveries": [ { "from": "node_A", - "claimed_said": "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq", - "cbor_hex": "a56164783545676379356c6b77797034346b756d6175636e6f366f63736a7a763434747832636637646335327a3636376d757871376d7334687165636172676fa464747970657573656e736f722e74656d70657261747572652e76316973656e736f725f69646f726f6f66746f702d61727261792d376d63656c736975735f6d696c6c69195bb36d72656164696e675f6672616d651a0002cfac6a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730875657374696d617465645f6d6963726f6a6f756c65730a6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" + "claimed_said": "Ek2atam3xjn73papa7bkh5ltoypfci4umrl3k5hoxohi6ajmpnu4q", + "cbor_hex": "a261647835456b326174616d33786a6e37337061706137626b68356c746f797066636934756d726c336b35686f786f686936616a6d706e753471616e01" }, { "from": "node_B", - "claimed_said": "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq", - "cbor_hex": "a56164783545676379356c6b77797034346b756d6175636e6f366f63736a7a763434747832636637646335327a3636376d757871376d7334687165636172676fa464747970657573656e736f722e74656d70657261747572652e76316973656e736f725f69646f726f6f66746f702d61727261792d376d63656c736975735f6d696c6c691a0001869f6d72656164696e675f6672616d651a0002cfac6a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730875657374696d617465645f6d6963726f6a6f756c65730a6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" + "claimed_said": "Ek2atam3xjn73papa7bkh5ltoypfci4umrl3k5hoxohi6ajmpnu4q", + "cbor_hex": "a261647835456b326174616d33786a6e37337061706137626b68356c746f797066636934756d726c336b35686f786f686936616a6d706e753471616e02" } ], "expected_results": [ { "outcome": "accepted", - "said": "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq" + "said": "Ek2atam3xjn73papa7bkh5ltoypfci4umrl3k5hoxohi6ajmpnu4q" }, { "outcome": "rejected", @@ -47,23 +50,26 @@ } ], "expected_held": [ - "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq" + { + "said": "Ek2atam3xjn73papa7bkh5ltoypfci4umrl3k5hoxohi6ajmpnu4q", + "cbor_hex": "a261647835456b326174616d33786a6e37337061706137626b68356c746f797066636934756d726c336b35686f786f686936616a6d706e753471616e01" + } ] }, { "name": "gossip-003-backfill-answer-verified", - "description": "node_B lacks the object behind said-004's SAID and asks for it by identity: any party holding an identifier can request the rest from any peer (treatise §21.4). node_A answers with said-005's object, which is well-formed and self-consistent but not the one requested, so it is refused. node_D's answer hashes to the requested SAID and is accepted. Spec v1 leaves gossip propagation rules to a later revision (treatise §21.6); what it does fix is that an answer is checked against the identifier that was asked for.", + "description": "node_B requests the object behind said-007's SAID: a party holding an identifier can request the object from any peer (treatise §21.4). node_A answers with said-008's object, which is genuine and self-consistent but not the one requested, and is refused. node_D's answer hashes to the requested SAID and is accepted (identity spec §2.6).", "receiver": "node_B", "deliveries": [ { "from": "node_A", - "claimed_said": "Ees7ph2gwaouwjro6nb5wpwdoxhlxyeannprp7bprskconubd6a5a", - "cbor_hex": "a56164783545797a796c697769676d72707a63726834656379776863326976716d64377965376f6477777361376c796562696763627a61687a6165636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c6573182f75657374696d617465645f6d6963726f6a6f756c657318386a70726f76656e616e6365a463616c6701697369676e61747572655840222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222226d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111116e69737375616e63655f6672616d651904006f6f776e6572736869705f636861696e82a663616c6701656672616d6519041a697369676e617475726558404444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444469746f5f7075626b6579582033333333333333333333333333333333333333333333333333333333333333336b66726f6d5f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117570726576696f75735f686973746f72795f68617368582024bef3e8d603a964c5de687b67d86eb9d77c100d6be2ff85f19284e6d023f03aa663616c6701656672616d65190438697369676e617475726558406666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666669746f5f7075626b6579582055555555555555555555555555555555555555555555555555555555555555556b66726f6d5f7075626b6579582033333333333333333333333333333333333333333333333333333333333333337570726576696f75735f686973746f72795f68617368582048dd9261e8f0836e26d963e2924a0441c2ad1620654cbe76159581d10775aa15" + "claimed_said": "Eze2x3zkvaaxvxu6hb75kidinrkmtczqu63b6qnzbg3g75e6hnwra", + "cbor_hex": "a261647835456b326174616d33786a6e37337061706137626b68356c746f797066636934756d726c336b35686f786f686936616a6d706e753471616e01" }, { "from": "node_D", - "claimed_said": "Ees7ph2gwaouwjro6nb5wpwdoxhlxyeannprp7bprskconubd6a5a", - "cbor_hex": "a561647835456573377068326777616f75776a726f366e6235777077646f78686c787965616e6e70727037627072736b636f6e7562643661356165636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730e75657374696d617465645f6d6963726f6a6f756c6573126a70726f76656e616e6365a463616c6701697369676e61747572655840222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222226d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111116e69737375616e63655f6672616d651904006f6f776e6572736869705f636861696e80" + "claimed_said": "Eze2x3zkvaaxvxu6hb75kidinrkmtczqu63b6qnzbg3g75e6hnwra", + "cbor_hex": "a261647835457a653278337a6b7661617876787536686237356b6964696e726b6d74637a717536336236716e7a6267336737356536686e777261616b582101d75a980182b10ab7d54bfed3c964073a0ee172f3daa62325af021a68f707511a" } ], "expected_results": [ @@ -73,46 +79,52 @@ }, { "outcome": "accepted", - "said": "Ees7ph2gwaouwjro6nb5wpwdoxhlxyeannprp7bprskconubd6a5a" + "said": "Eze2x3zkvaaxvxu6hb75kidinrkmtczqu63b6qnzbg3g75e6hnwra" } ], "expected_held": [ - "Ees7ph2gwaouwjro6nb5wpwdoxhlxyeannprp7bprskconubd6a5a" + { + "said": "Eze2x3zkvaaxvxu6hb75kidinrkmtczqu63b6qnzbg3g75e6hnwra", + "cbor_hex": "a261647835457a653278337a6b7661617876787536686237356b6964696e726b6d74637a717536336236716e7a6267336737356536686e777261616b582101d75a980182b10ab7d54bfed3c964073a0ee172f3daa62325af021a68f707511a" + } ] }, { "name": "gossip-004-duplicate-delivery-deduplicated", - "description": "The same object reaches node_B from two peers. The second delivery is a duplicate of a SAID already held and changes nothing: the overlay deduplicates by content address (treatise §11.4).", + "description": "The same object reaches node_B from two peers. The second delivery verifies and is a duplicate of a SAID already held, so it changes nothing: the overlay deduplicates by content address (treatise §11.4).", "receiver": "node_B", "deliveries": [ { "from": "node_A", - "claimed_said": "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq", - "cbor_hex": "a5616478354574626c613563346e61787372657861666c787a72686d376672706b626463656769746b67686f6c663663636d74366977706f787165636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730c75657374696d617465645f6d6963726f6a6f756c65730f6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" + "claimed_said": "E4duyldlhqjm25aiznrd2dxsz6uxfhx6vyqj2itotg3hjahvqakiq", + "cbor_hex": "a16164783545346475796c646c68716a6d323561697a6e7264326478737a367578666878367679716a3269746f746733686a61687671616b6971" }, { "from": "node_C", - "claimed_said": "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq", - "cbor_hex": "a5616478354574626c613563346e61787372657861666c787a72686d376672706b626463656769746b67686f6c663663636d74366977706f787165636172676fa364747970656c7573642e636172676f2e76316d6973737565725f7075626b6579582011111111111111111111111111111111111111111111111111111111111111117364656e6f6d696e6174696f6e5f6d6963726f731a05f5e1006a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730c75657374696d617465645f6d6963726f6a6f756c65730f6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" + "claimed_said": "E4duyldlhqjm25aiznrd2dxsz6uxfhx6vyqj2itotg3hjahvqakiq", + "cbor_hex": "a16164783545346475796c646c68716a6d323561697a6e7264326478737a367578666878367679716a3269746f746733686a61687671616b6971" } ], "expected_results": [ { "outcome": "accepted", - "said": "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq" + "said": "E4duyldlhqjm25aiznrd2dxsz6uxfhx6vyqj2itotg3hjahvqakiq" }, { "outcome": "duplicate", - "said": "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq" + "said": "E4duyldlhqjm25aiznrd2dxsz6uxfhx6vyqj2itotg3hjahvqakiq" } ], "expected_held": [ - "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq" + { + "said": "E4duyldlhqjm25aiznrd2dxsz6uxfhx6vyqj2itotg3hjahvqakiq", + "cbor_hex": "a16164783545346475796c646c68716a6d323561697a6e7264326478737a367578666878367679716a3269746f746733686a61687671616b6971" + } ] }, { "name": "gossip-005-non-canonical-copy-rejected", - "description": "said-009's bytes reach node_B announced under said-008's SAID, the SAID a re-encoding receiver would compute for them. The bytes are not canonical, so a conforming receiver refuses them (identity spec §2.5). Accepting them would let a second byte string circulate under said-008's SAID.", + "description": "said-009's bytes reach node_B announced under said-008's SAID, the SAID a re-encoding receiver would compute for them. The bytes are not canonical, so node_B refuses them (identity spec §2.5, §2.6). Accepting them would let a second byte string circulate under said-008's SAID.", "receiver": "node_B", "deliveries": [ { @@ -131,18 +143,18 @@ }, { "name": "gossip-006-substitute-arriving-first-rejected", - "description": "gossip-002 in the other order: the altered object reaches node_C first and is refused, then said-003's object arrives and is accepted. Arrival order decides nothing; the content does.", + "description": "gossip-002 in the other order: the altered object reaches node_C first and is refused, then said-008's object arrives and is accepted. Arrival order decides nothing; the content does.", "receiver": "node_C", "deliveries": [ { "from": "node_B", - "claimed_said": "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq", - "cbor_hex": "a56164783545676379356c6b77797034346b756d6175636e6f366f63736a7a763434747832636637646335327a3636376d757871376d7334687165636172676fa464747970657573656e736f722e74656d70657261747572652e76316973656e736f725f69646f726f6f66746f702d61727261792d376d63656c736975735f6d696c6c691a0001869f6d72656164696e675f6672616d651a0002cfac6a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730875657374696d617465645f6d6963726f6a6f756c65730a6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" + "claimed_said": "Ek2atam3xjn73papa7bkh5ltoypfci4umrl3k5hoxohi6ajmpnu4q", + "cbor_hex": "a261647835456b326174616d33786a6e37337061706137626b68356c746f797066636934756d726c336b35686f786f686936616a6d706e753471616e02" }, { "from": "node_A", - "claimed_said": "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq", - "cbor_hex": "a56164783545676379356c6b77797034346b756d6175636e6f366f63736a7a763434747832636637646335327a3636376d757871376d7334687165636172676fa464747970657573656e736f722e74656d70657261747572652e76316973656e736f725f69646f726f6f66746f702d61727261792d376d63656c736975735f6d696c6c69195bb36d72656164696e675f6672616d651a0002cfac6a6a6f756c655f636f7374a2746d656173757265645f6d6963726f6a6f756c65730875657374696d617465645f6d6963726f6a6f756c65730a6a70726f76656e616e6365f66f6f776e6572736869705f636861696e80" + "claimed_said": "Ek2atam3xjn73papa7bkh5ltoypfci4umrl3k5hoxohi6ajmpnu4q", + "cbor_hex": "a261647835456b326174616d33786a6e37337061706137626b68356c746f797066636934756d726c336b35686f786f686936616a6d706e753471616e01" } ], "expected_results": [ @@ -152,11 +164,14 @@ }, { "outcome": "accepted", - "said": "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq" + "said": "Ek2atam3xjn73papa7bkh5ltoypfci4umrl3k5hoxohi6ajmpnu4q" } ], "expected_held": [ - "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq" + { + "said": "Ek2atam3xjn73papa7bkh5ltoypfci4umrl3k5hoxohi6ajmpnu4q", + "cbor_hex": "a261647835456b326174616d33786a6e37337061706137626b68356c746f797066636934756d726c336b35686f786f686936616a6d706e753471616e01" + } ] } ] diff --git a/smart-byte/conformance/v1/lockstep_vectors.json b/smart-byte/conformance/v1/lockstep_vectors.json index a5be91713..5bf96d8db 100644 --- a/smart-byte/conformance/v1/lockstep_vectors.json +++ b/smart-byte/conformance/v1/lockstep_vectors.json @@ -1,10 +1,16 @@ [ { "name": "lockstep-001-unanimous-commit", - "description": "All four nodes report the same post-frame state hash. The threshold is floor(2·4/3)+1 = 3, so the frame commits, every node's outcome is a clean commit (treatise §13.3), and nothing is logged as dissent. State hashes in these vectors are opaque 32-byte values compared only for equality: §9.4 does not pin the state's serialization, so the commit rule is tested over the hashes nodes report.", + "description": "All four nodes report the post-frame state hash they computed, and all agree. The threshold is floor(2·4/3)+1 = 3, so the frame commits, every node's outcome is a clean commit (treatise §13.3), and no dissent is logged. State hashes are opaque 32-byte values compared for equality. Each node's outcome comes from its own view: the hashes its peers reported plus the one it computed itself (§13.3, §13.4).", "node_count": 4, "frame_number": 184237, - "node_hashes": { + "node_reported_hashes": { + "node_1": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "node_2": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "node_3": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "node_4": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + }, + "node_computed_hashes": { "node_1": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "node_2": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "node_3": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", @@ -19,14 +25,25 @@ "node_3": "clean_commit", "node_4": "clean_commit" }, - "expected_dissent": [] + "expected_dissent_logs": { + "node_1": [], + "node_2": [], + "node_3": [], + "node_4": [] + } }, { "name": "lockstep-002-supermajority-commit", - "description": "Three of four nodes agree and node_4 reports another hash. The threshold of 3 is met and the frame commits at the majority hash. Nodes 1–3 commit with dissent and write one dissent-log entry naming node_4 (treatise §14.1). node_4's own outcome is minority divergence: it halts, and writes nothing.", + "description": "Three of four nodes agree. node_4's state diverged, and it reports the hash it computed. The threshold of 3 is met: nodes 1–3 commit with dissent, and each writes one dissent-log entry naming node_4 (treatise §14.1). node_4 sees a supermajority on a hash that is not its own — minority divergence — so it halts and writes nothing.", "node_count": 4, "frame_number": 184238, - "node_hashes": { + "node_reported_hashes": { + "node_1": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "node_2": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "node_3": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "node_4": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd" + }, + "node_computed_hashes": { "node_1": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "node_2": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "node_3": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", @@ -41,23 +58,52 @@ "node_3": "commit_with_dissent", "node_4": "minority_divergence" }, - "expected_dissent": [ - { - "frame": 184238, - "dissenting_peers": [ - "node_4" - ], - "local_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", - "supermajority_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" - } - ] + "expected_dissent_logs": { + "node_1": [ + { + "frame": 184238, + "dissenting_peers": [ + "node_4" + ], + "local_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "supermajority_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + } + ], + "node_2": [ + { + "frame": 184238, + "dissenting_peers": [ + "node_4" + ], + "local_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "supermajority_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + } + ], + "node_3": [ + { + "frame": 184238, + "dissenting_peers": [ + "node_4" + ], + "local_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "supermajority_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + } + ], + "node_4": [] + } }, { "name": "lockstep-003-threshold-not-met", - "description": "No hash is reported by more than two of the four nodes. None reaches the threshold of 3, so the frame does not commit and every node's outcome is a cluster split (§13.3). No dissent is recorded: dissent is divergence from a committed hash (§14.2), and there is none.", + "description": "No hash is reported by more than two of the four nodes. None reaches the threshold of 3, so the frame does not commit and every node's outcome is a cluster split (§13.3). No dissent is logged: dissent is divergence from a committed hash (§14.2), and there is none.", "node_count": 4, "frame_number": 184239, - "node_hashes": { + "node_reported_hashes": { + "node_1": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", + "node_2": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", + "node_3": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", + "node_4": "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee" + }, + "node_computed_hashes": { "node_1": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", "node_2": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", "node_3": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", @@ -72,14 +118,24 @@ "node_3": "cluster_split", "node_4": "cluster_split" }, - "expected_dissent": [] + "expected_dissent_logs": { + "node_1": [], + "node_2": [], + "node_3": [], + "node_4": [] + } }, { - "name": "lockstep-004-three-nodes-one-liar-split", - "description": "The n = 3 exception (treatise §13.2). The threshold is 3, so one node reporting a different hash leaves 2 of the 3 votes the frame needs. The cluster detects the divergence but cannot tolerate it: no commit, and every node halts with a cluster split.", + "name": "lockstep-004-three-nodes-one-diverged-split", + "description": "The n = 3 exception (treatise §13.2). The threshold is 3, so one node whose state diverged leaves 2 of the 3 votes the frame needs. The cluster detects the divergence but cannot tolerate it: every node halts with a cluster split.", "node_count": 3, "frame_number": 184240, - "node_hashes": { + "node_reported_hashes": { + "node_1": "1111111111111111111111111111111111111111111111111111111111111111", + "node_2": "1111111111111111111111111111111111111111111111111111111111111111", + "node_3": "2222222222222222222222222222222222222222222222222222222222222222" + }, + "node_computed_hashes": { "node_1": "1111111111111111111111111111111111111111111111111111111111111111", "node_2": "1111111111111111111111111111111111111111111111111111111111111111", "node_3": "2222222222222222222222222222222222222222222222222222222222222222" @@ -92,14 +148,27 @@ "node_2": "cluster_split", "node_3": "cluster_split" }, - "expected_dissent": [] + "expected_dissent_logs": { + "node_1": [], + "node_2": [], + "node_3": [] + } }, { "name": "lockstep-005-seven-nodes-two-dissenters", - "description": "Seven nodes, threshold 5: five agree and two report hashes of their own. The frame commits. The dissent-log entry names both dissenters in ascending order, so every honest node writes the same entry (§14.2).", + "description": "Seven nodes, threshold 5: five agree and two diverged nodes report hashes of their own. The frame commits. The dissent entry names both dissenters in ascending order, so every committing node writes the same entry (§14.2).", "node_count": 7, "frame_number": 184241, - "node_hashes": { + "node_reported_hashes": { + "node_1": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "node_2": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "node_3": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "node_4": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "node_5": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "node_6": "6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b", + "node_7": "7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c7c" + }, + "node_computed_hashes": { "node_1": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", "node_2": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", "node_3": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", @@ -120,16 +189,153 @@ "node_6": "minority_divergence", "node_7": "minority_divergence" }, - "expected_dissent": [ - { - "frame": 184241, - "dissenting_peers": [ - "node_6", - "node_7" - ], - "local_hash": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", - "supermajority_hash": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a" - } - ] + "expected_dissent_logs": { + "node_1": [ + { + "frame": 184241, + "dissenting_peers": [ + "node_6", + "node_7" + ], + "local_hash": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "supermajority_hash": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a" + } + ], + "node_2": [ + { + "frame": 184241, + "dissenting_peers": [ + "node_6", + "node_7" + ], + "local_hash": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "supermajority_hash": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a" + } + ], + "node_3": [ + { + "frame": 184241, + "dissenting_peers": [ + "node_6", + "node_7" + ], + "local_hash": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "supermajority_hash": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a" + } + ], + "node_4": [ + { + "frame": 184241, + "dissenting_peers": [ + "node_6", + "node_7" + ], + "local_hash": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "supermajority_hash": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a" + } + ], + "node_5": [ + { + "frame": 184241, + "dissenting_peers": [ + "node_6", + "node_7" + ], + "local_hash": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "supermajority_hash": "5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a" + } + ], + "node_6": [], + "node_7": [] + } + }, + { + "name": "lockstep-006-transport-liar-is-logged", + "description": "Treatise §13.4: node_4 computed the honest hash but broadcast another. Its peers see 3-of-4 agreement and record node_4 in their dissent logs. node_4 tallies its own honest hash, sees unanimity and commits cleanly: its lie surfaces only in its peers' logs, which is why the dissent log is the substrate's misbehaviour signal (§14).", + "node_count": 4, + "frame_number": 184242, + "node_reported_hashes": { + "node_1": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "node_2": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "node_3": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "node_4": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd" + }, + "node_computed_hashes": { + "node_1": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "node_2": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "node_3": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "node_4": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + }, + "expected_threshold": 3, + "expected_commit": true, + "expected_state_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "expected_outcomes": { + "node_1": "commit_with_dissent", + "node_2": "commit_with_dissent", + "node_3": "commit_with_dissent", + "node_4": "clean_commit" + }, + "expected_dissent_logs": { + "node_1": [ + { + "frame": 184242, + "dissenting_peers": [ + "node_4" + ], + "local_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "supermajority_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + } + ], + "node_2": [ + { + "frame": 184242, + "dissenting_peers": [ + "node_4" + ], + "local_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "supermajority_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + } + ], + "node_3": [ + { + "frame": 184242, + "dissenting_peers": [ + "node_4" + ], + "local_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "supermajority_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + } + ], + "node_4": [] + } + }, + { + "name": "lockstep-007-three-node-transport-liar", + "description": "§13.2 and §13.4 together. At n = 3 a node that broadcasts a hash other than the one it computed splits the honest pair, who halt. The liar sees its own honest hash agree with both peers and commits. The reported hashes reach no supermajority, so the frame's public record is no commit.", + "node_count": 3, + "frame_number": 184243, + "node_reported_hashes": { + "node_1": "1111111111111111111111111111111111111111111111111111111111111111", + "node_2": "1111111111111111111111111111111111111111111111111111111111111111", + "node_3": "2222222222222222222222222222222222222222222222222222222222222222" + }, + "node_computed_hashes": { + "node_1": "1111111111111111111111111111111111111111111111111111111111111111", + "node_2": "1111111111111111111111111111111111111111111111111111111111111111", + "node_3": "1111111111111111111111111111111111111111111111111111111111111111" + }, + "expected_threshold": 3, + "expected_commit": false, + "expected_state_hash": null, + "expected_outcomes": { + "node_1": "cluster_split", + "node_2": "cluster_split", + "node_3": "clean_commit" + }, + "expected_dissent_logs": { + "node_1": [], + "node_2": [], + "node_3": [] + } } ] diff --git a/smart-byte/conformance/v1/regeneration.md b/smart-byte/conformance/v1/regeneration.md index 261093126..0142c418c 100644 --- a/smart-byte/conformance/v1/regeneration.md +++ b/smart-byte/conformance/v1/regeneration.md @@ -57,12 +57,14 @@ The conformance gate runs the second implementation on every pack change. spells (`signed_message_hex`), not its 53-character text. A positive vector's `signature` is the deterministic Ed25519 signature by `signing_key_seed`; a negative vector is refused with its `error` class. -4. **Lockstep vectors.** The reported `node_hashes` are tallied against the - threshold floor(2n/3)+1 (treatise §13.1), giving each node its §13.3 outcome - and the §14.1 dissent-log entry the committing nodes write. -5. **Gossip vectors.** Each delivery goes through the intake rule the vectors - describe: bytes must be canonical, must hash to the SAID they state, and that - SAID must be the one they arrived under; a SAID already held is a duplicate. +4. **Lockstep vectors.** The hashes nodes reported are tallied against the + threshold floor(2n/3)+1 (treatise §13.1) for the frame's public record. + Each node's outcome (§13.3) and dissent log (§14.1) come from its own + view: its peers' reported hashes plus the hash it computed, so a node that + lies on the wire (§13.4) is modelled exactly as the treatise describes. +5. **Gossip vectors.** Each delivery goes through `smart-byte-core`'s inbox, + which implements identity spec §2.6: canonical bytes, then the SAID they + state, then the SAID they arrived under, and only then deduplication. ## Invariant: cross-vector consistency @@ -93,111 +95,110 @@ None of them signed any SAID. All five canonicalisation encodings were wrong, and two of them were odd-length hex that could not be decoded at all. Every value is now computed, and checked twice (see above). -**The pack was regenerated in place, not by supersession.** The pack is DRAFT -(`../VERSIONS.md`), and in-place regeneration was the stated plan for its -placeholders. No vector replaced here could have been passed by any -implementation, so no implementation can have pinned one. The supersession rule -(`../README.md`, `VERSIONS.md` rule 4) governs v1 from its freeze onward. +**Regenerated in place.** v1 is DRAFT, and a DRAFT pack is regenerated in +place; supersession governs a frozen pack (`../VERSIONS.md` rule 4, now +explicit about this). A few draft vectors did hold values an implementation +could pass — lockstep-001's tally over identical hashes, and the sig-004 and +sig-005 negatives. They keep their names and subjects here. **Inputs that had to change before anything could be computed:** - JSON has no byte-string type, and the draft wrote byte values as hex text, which is indistinguishable from text. Byte strings are now written - `{"$bytes": ""}` (see `../README.md`). + `{"$bytes": ""}`, and `$bytes` is reserved (`../README.md`). - SAID vectors: the SAID field is `d`, holding the 53-`#` placeholder (identity spec §2.1, §2.2). The draft used a `said` member holding ``. The object member is now `input` rather than `envelope`, and the `version` - member is gone: treatise §8.1 gives an envelope five fields, and the - algorithm identifier carries version information (`VERSIONS.md` rule 5). -- `said-005`'s `previous_history_hash` values were SAID-shaped filler. They are - now computed per treatise §10.1. The draft also gave said-005 the same SAID as - said-004. A §2 SAID covers the whole object, so they differ, and the - description now says why a byte's identity is fixed at its origin instead. + member is gone. +- `said-005`'s `previous_history_hash` values were SAID-shaped filler. They + are now computed, and said-005's SAID is its own. The draft gave it + said-004's SAID, but a SAID covers the whole object it is derived over. - `canon-003`'s byte string was given as base64 text; it is now bytes. **Vectors whose meaning contradicted the spec, corrected (and renamed where the name stated the wrong rule):** - `canon-004-deterministic-float-encoding` → `canon-004-floats-forbidden`. The - draft encoded floats in their shortest exact width, but identity spec §2.3 - forbids floats. canon-004 is now a negative vector, and canon-016/017 cover - the other widths. -- `lockstep-003`. The draft logged dissent for all four nodes of a frame that - did not commit. Treatise §14.2 writes the dissent log only on a commit with - dissent, so there is no entry. Every lockstep dissent entry now has §14.1's - shape: one entry per frame, holding the dissenting peer set, the local hash - and the supermajority hash. Each vector also states its threshold and every - node's §13.3 outcome. + draft encoded floats in their shortest exact width; identity spec §2.3 + forbids floats. canon-016/017 cover the other widths. +- `lockstep-003` logged dissent for all four nodes of a frame that did not + commit. Treatise §14.2 writes the dissent log only on a commit with dissent. +- Lockstep vectors now separate the hash a node broadcast from the one it + computed, and give each node's outcome and dissent log from its own view. + The draft's single view called a node a "liar" while modelling a node whose + state diverged; treatise §13.4's liar commits in its own view while its + peers log it, which lockstep-006 and lockstep-007 now pin. lockstep-004 is + renamed to say what it models: a diverged node. - Lockstep vectors no longer carry `ordered_transitions`. They were never - inputs to the commit rule, which is decided over the hashes nodes report - (§9.4 does not pin the state's serialization), and their "signatures" were - filler. + inputs to the commit rule, and their "signatures" were filler. - `gossip-002-said-conflict-rejected` keeps its name but not its rule. The - draft settled a "collision" by keeping whichever object arrived first. Two - different canonical objects cannot share a SAID short of a BLAKE3 collision, - so the newcomer is refused because it does not hash to the SAID it states, - whatever the arrival order (gossip-006 is the reverse order). + draft kept whichever object arrived first. Two different canonical objects + cannot share a SAID short of a BLAKE3 collision, so the newcomer is refused + because it does not hash to the SAID it states, in either arrival order + (gossip-006). Verification precedes deduplication (identity spec §2.6). - `gossip-003-sequence-gap-detection` → `gossip-003-backfill-answer-verified`. The draft asserted a 256-frame backfill limit, request targets and alert - classes that no spec text defines; treatise §21.6 defers gossip rules to a - later revision. The replacement tests what the spec does fix: an answer to a - request by identity is checked against that identity (§21.4, §10.2 step 1). -- `gossip-001` no longer asserts publish/observation frames or round counts: - gossip is not frame-synchronous in v1. -- Signature vectors: sig-003 and sig-004 now share the error class - `signature_invalid`. A verifier cannot tell "wrong SAID" from "wrong key"; - both are the same failed equation. sig-004's key is now derived from its - seed; the draft's key was derived from nothing. sig-005's class is - `malformed_signature`, and its timing claim is gone: a vector cannot test - timing, and this one requires the structural failure to be reported - distinctly. + classes that no spec text defines. The replacement tests what the spec + fixes: an answer to a request by SAID is checked against that SAID. +- Gossip vectors carry generic SAID-addressed objects, not envelope-shaped + ones. A node that also checks a byte's provenance (treatise §8.3) must + refuse an envelope with null or filler provenance, so such deliveries could + not be MUST-pass. §2.6 applies to every SAID-addressed object. +- `gossip-001` no longer asserts frames or round counts — gossip is not + frame-synchronous — and `expected_held` now carries the bytes held. +- Signature vectors: sig-003 and sig-004 share the class `signature_invalid`. + Both are the same failed equation, and no verifier can tell them apart. + sig-004's key is derived from its seed, and sig-005's class is + `malformed_signature`. -**Vectors added:** canon-006 to canon-019 (each §2.3 rule, accepted and -refused); said-006 to said-011 (the suite identity spec §2.5 names: the -empty-map case, the one-field case, the worked example, two adversarial -non-canonical encodings, and a changed-content mismatch); sig-006 (a -non-canonical S, RFC 8032 §5.1.7); lockstep-004 (the n = 3 exception, §13.2); -lockstep-005 (two dissenters); gossip-004 to gossip-006. +**Vectors added:** -**Spec text changed with it:** +- canon-006 to canon-020: each §2.3 rule, accepted and refused. +- said-006 to said-011: the suite identity spec §2.5 names. +- sig-006 (a non-canonical S) and sig-007 (a key spelled with y ≥ p). +- lockstep-004 to lockstep-007. +- gossip-004 to gossip-006. -- Identity spec §2.4 printed an invented digest and SAID, and counted the - encoding as 62 bytes; it is 61. It now prints said-008's values. -- §2.5 names the vectors. -- §6 pointed at modules that did not exist. It now says where each part lives, - including that the key-event log does not yet derive §2 SAIDs. +**Spec text changed with it (identity spec):** + +- §2.4 printed an invented digest and SAID, and counted 62 bytes; it now + prints said-008's values for its 61 bytes. +- §2.1, §2.3, §2.4 and §4 step 1 told a verifier to re-encode an object + canonically, which is exactly what §2.5's adversarial vector forbids. They + now require received bytes to be canonical already. +- §2.3 now forbids simple values other than false, true and null. +- §2.6 is new: receiving SAID-addressed objects. +- §3.3 and §3.4 now say a key is a byte string. +- §2.5 and §6 now name the vectors and the modules. **Reference implementation changed with it:** -- `smart-byte-core` gained canonical CBOR with a strict decoder, the §2 SAID - text form with derivation, verification and per-key SAIDs, and SAID signature - verification. -- `smart-byte-lockstep` gained the §13/§14 tally. `Cluster::step` uses it and - now keeps a dissent log. -- `smart-byte-libp2p` now checks received envelopes against the ids they state - on every gossip receive path, and checks a fetch's answer against the SAID - that was requested. Before, neither path checked, so a peer could answer a - fetch for one envelope with another. +- `smart-byte-core`: + - canonical CBOR, with a strict decoder that never allocates from a + declared count and reports the first defect in stream order; + - §2 SAIDs and the §2.6 inbox; + - SAID signature verification, with RFC 8032-strict key decoding; + - `Envelope::from_cbor`, which accepts only an envelope's own encoding. +- `smart-byte-lockstep`: the §13/§14 tally and each node's own view. + `Cluster::step` keeps a dissent log. +- `smart-byte-libp2p`: + - gossip is validated before it is forwarded, so a forged or padded + envelope stops at the first honest node; + - a fetch's answer must be the envelope requested; + - one bad message no longer ends a wait. +- `smart-byte-keri-witness`: the verifier rejects `rec` by default, as spec + v1 requires. Recovery is an explicit opt-in. -**Found and not fixed here.** These are normative or larger decisions: +**Checked whole.** Both checkers require every file to be non-empty, every +vector the identity spec cites to be present, and every README error class +to be exercised — and no other. A pack cannot pass by losing vectors. The +Python checker raises its own failures rather than using `assert`, so +`python -O` cannot switch it off. Both refuse JSON that names a member twice. -1. The reference `Envelope` type and the key-event log in - `smart-byte-keri-witness` derive identifiers with a general-purpose CBOR - serializer and a zeroed placeholder, not §2, so their ids are not §2 SAIDs. - Migrating them changes every identifier they produce. For `Envelope` it also - means first settling its wire schema against treatise §8: its wall-clock - issuance time and string-tagged cargo diverge from §8.3, §8.5 and §8.7. -2. Identity spec §4 step 1 replaces only `d` with the placeholder, but an - inception event's `i` is derived from its own `d` (§3.1), so the inception - SAID cannot be computed as written. The reference key-event log blanks `i` - too. The spec should say what to do. -3. §3.4 writes the per-key object's `k` as `""`, while §3.3 says a key is - prefixed with a one-byte algorithm identifier. said-007 carries `k` as a - byte string. If text was intended, said-007 changes. -4. The state serialization behind a post-frame hash (§9.4 step 6) and the - transition payload behind a history hash (§10.1) are not pinned, so no - vector can pin a real state hash or history hash. said-005 illustrates the - chain step with the payload taken as the transition's canonical encoding. +**Next:** envelope and key-event-log identifiers move to §2 SAIDs. The +envelope's wire format and signing preimages get a spec of their own, and an +inception event placeholders `i` with `d`, because it derives its controller +AID from its own SAID. ### 2026-09-21 — spec reference corrected; no wire-format change diff --git a/smart-byte/conformance/v1/said_vectors.json b/smart-byte/conformance/v1/said_vectors.json index 564ced49f..7f580dfa7 100644 --- a/smart-byte/conformance/v1/said_vectors.json +++ b/smart-byte/conformance/v1/said_vectors.json @@ -1,7 +1,7 @@ [ { "name": "said-001-empty-envelope", - "description": "The smallest envelope-shaped object (treatise §8.1): no provenance, no cargo, no ownership transitions, zero joule cost. It exercises the whole pipeline — placeholder, canonical CBOR, BLAKE3-256, `E` + base32 — on an object with no application data.", + "description": "A nested object with an envelope's top-level members — provenance, ownership chain, cargo, joule cost — all empty or zero. It exercises the whole pipeline (placeholder, canonical CBOR, BLAKE3-256, `E` + base32) on an object with no application data.", "said_field": "d", "input": { "d": "#####################################################", @@ -20,7 +20,7 @@ }, { "name": "said-002-usd-cargo", - "description": "An envelope-shaped object carrying a USD cargo of 100.00 (100000000 micros) from issuer key 0x11…11. Cargo is opaque to the substrate; here it is a map whose members include a byte-string key.", + "description": "The same object carrying a cargo map: a USD amount of 100.00 (100000000 micros) and a 32-byte issuer key. Members of mixed types — text, integer, byte string — all sort by their encoded keys.", "said_field": "d", "input": { "d": "#####################################################", @@ -45,7 +45,7 @@ }, { "name": "said-003-custom-cargo", - "description": "The same shape carrying a non-money cargo: a temperature reading. The pipeline treats it exactly as it treats said-002 — cargo opacity is the property under test.", + "description": "The same shape carrying a temperature reading as cargo. What a member means never affects its encoding: this cargo goes through the pipeline exactly as said-002's does.", "said_field": "d", "input": { "d": "#####################################################", @@ -69,7 +69,7 @@ }, { "name": "said-004-with-provenance", - "description": "An envelope-shaped object with a provenance attestation: algorithm identifier 0x01 (Ed25519), issuer key, issuance frame and signature. The SAID covers the provenance. The key and signature are fixed filler bytes: this vector tests SAID derivation, not signature validity, which signature_vectors.json covers.", + "description": "The same shape with a provenance map: algorithm identifier 0x01 (Ed25519), issuer key, issuance frame and signature. The SAID covers every member. The key and signature are fixed filler bytes: this vector tests SAID derivation over them, not their validity.", "said_field": "d", "input": { "d": "#####################################################", @@ -103,7 +103,7 @@ }, { "name": "said-005-with-ownership-transitions", - "description": "said-004's byte after two transfers, as fetched at frame 1100. The first transition's previous_history_hash is the digest of said-004's SAID, because a byte's identity is its initial history hash (treatise §10.1); the second is blake3 of that digest followed by the first transition's canonical encoding, illustrating §10.1's chain step (spec v1 does not pin the payload's serialization). This object's SAID differs from said-004's: a SAID is a function of the whole object, which is why a byte's identity is fixed at its origin (treatise §8.2) rather than recomputed as transitions accrue.", + "description": "said-004's object with two ownership transitions appended and a larger joule cost: an array of maps holding byte strings. The first transition's previous_history_hash is the digest of said-004's SAID; the second is blake3 of that digest followed by the first transition's canonical encoding. The SAID differs from said-004's because a SAID covers the whole object it is derived over. A byte's identity is a different quantity — the hash of its origin attestation (treatise §8.2) — and does not change as transitions accrue.", "said_field": "d", "input": { "d": "#####################################################", @@ -182,7 +182,7 @@ }, { "name": "said-007-one-field-per-key", - "description": "The one-field case: identity spec §3.4's per-key object {d, k}, whose SAID is a next-key commitment. k is the algorithm identifier 0x01 (Ed25519) followed by the public key of RFC 8032's first test seed, carried as a byte string.", + "description": "The one-field case: identity spec §3.4's per-key object {d, k}, whose SAID is a next-key commitment. k is a byte string: the algorithm identifier 0x01 (Ed25519) followed by the public key of RFC 8032's first test seed (§3.3).", "said_field": "d", "input": { "d": "#####################################################", diff --git a/smart-byte/conformance/v1/signature_vectors.json b/smart-byte/conformance/v1/signature_vectors.json index 6437dabe3..979433acb 100644 --- a/smart-byte/conformance/v1/signature_vectors.json +++ b/smart-byte/conformance/v1/signature_vectors.json @@ -1,7 +1,7 @@ [ { "name": "sig-001-known-key-known-said", - "description": "Sign said-002's SAID with the key of RFC 8032's first test seed. The message is the 32 digest bytes the SAID spells (signed_message_hex), not its 53-character text. Ed25519 is deterministic, so every implementation produces exactly this signature.", + "description": "Ed25519 (RFC 8032) is deterministic: signing said-002's SAID with the key of RFC 8032's first test seed gives exactly this signature. The message is the 32 digest bytes the SAID spells (signed_message_hex), not its 53-character text.", "envelope_said": "Etbla5c4naxsrexaflxzrhm7frpkbdcegitkgholf6ccmt6iwpoxq", "signed_message_hex": "98560e8b8d05e5125c055df313b3e58bd411888644d463b965f084c9f9167baf", "signing_key_seed": "9d61b19deffd5a60ba844af492ec2cc44449c5697b326919703bac031cae7f60", @@ -23,7 +23,7 @@ }, { "name": "sig-003-tampered-envelope", - "description": "sig-002's signature presented with said-003's SAID: an adversary swaps the envelope and forwards the original signature. The signature is well-formed but binds another SAID, so verification MUST fail.", + "description": "sig-002's signature presented with said-003's SAID: an adversary swaps the object and forwards the original signature. The signature is well-formed but binds another SAID, so verification MUST fail.", "envelope_said": "Egcy5lkwyp44kumaucno6ocsjzv44tx2cf7dc52z667muxq7ms4hq", "signed_message_hex": "30b1d5aad87f38aa3014135de70a49cd79c9df422fc62eeb3ef7d94bc3ec970f", "signing_key_seed": "4ccd089b28ff96da9db6c346ec114e0f5b8a319f35aba624da8cf6ed4fb8a6fb", @@ -34,7 +34,7 @@ }, { "name": "sig-004-wrong-key", - "description": "sig-002's signature over said-001's SAID, presented with another key (the one the all-zero seed derives). The signature is valid for the key that made it, but that is not this key, so verification MUST fail: a valid signature from a key that is not the byte's owner.", + "description": "sig-002's signature over said-001's SAID, presented with another key — the one the all-zero seed derives. The signature is valid for the key that made it, which is not this one, so verification MUST fail.", "envelope_said": "E64kxoz2b4kukifl5lbhnndcfos2bqxdjhp5wgx7zh35erpsv243a", "signed_message_hex": "f715776741e2a8a4157d584ed68c4574b4185c693bfb635ff93efa48be55d736", "signing_key_seed": "0000000000000000000000000000000000000000000000000000000000000000", @@ -56,7 +56,7 @@ }, { "name": "sig-006-non-canonical-s-rejected", - "description": "sig-002's signature with its scalar S replaced by S + L: the same scalar, written outside [0, L). RFC 8032 §5.1.7 requires refusing it. Accepting it would make signatures malleable, giving a second valid-looking signature for one message and key.", + "description": "sig-002's signature with its scalar S replaced by S + L: the same scalar, written outside [0, L). RFC 8032 §5.1.7 requires refusing it; accepting it would make signatures malleable, giving a second valid-looking signature for one message and key.", "envelope_said": "E64kxoz2b4kukifl5lbhnndcfos2bqxdjhp5wgx7zh35erpsv243a", "signed_message_hex": "f715776741e2a8a4157d584ed68c4574b4185c693bfb635ff93efa48be55d736", "signing_key_seed": "4ccd089b28ff96da9db6c346ec114e0f5b8a319f35aba624da8cf6ed4fb8a6fb", @@ -64,5 +64,16 @@ "signature": "ce9b9e7185967276bebe6a9c12f3570d821bcb7184e9c703bd278c1fa44ece65adcf51656f61522235552e76d315359f230af3156da0440cfaf5d4acd7123516", "expected_verify": false, "error": "signature_invalid" + }, + { + "name": "sig-007-non-canonical-key-rejected", + "description": "A verifying key spelled with y = p + 3: an out-of-range encoding of the valid, large-order point with y = 3. RFC 8032 §5.1.3 requires decoding to fail when y is not below p, so the key is refused as malformed before any verification runs — one key has one spelling. There is no seed: no key pair has this spelling.", + "envelope_said": "E64kxoz2b4kukifl5lbhnndcfos2bqxdjhp5wgx7zh35erpsv243a", + "signed_message_hex": "f715776741e2a8a4157d584ed68c4574b4185c693bfb635ff93efa48be55d736", + "signing_key_seed": null, + "verifying_key": "f0ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff7f", + "signature": "ce9b9e7185967276bebe6a9c12f3570d821bcb7184e9c703bd278c1fa44ece65c0fb5b0855fe3fca5eb836d3f41b568a230af3156da0440cfaf5d4acd7123506", + "expected_verify": false, + "error": "malformed_verifying_key" } ] diff --git a/smart-byte/smart-byte-rs/Cargo.lock b/smart-byte/smart-byte-rs/Cargo.lock index 611532150..0b5924726 100644 --- a/smart-byte/smart-byte-rs/Cargo.lock +++ b/smart-byte/smart-byte-rs/Cargo.lock @@ -5947,6 +5947,7 @@ name = "smart-byte-conformance" version = "0.1.0" dependencies = [ "hex", + "serde", "serde_json", "smart-byte-core", "smart-byte-lockstep", diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/Cargo.toml b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/Cargo.toml index 0f23acb9f..345b80b49 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/Cargo.toml +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/Cargo.toml @@ -11,6 +11,7 @@ publish = false [dependencies] smart-byte-core = { workspace = true } smart-byte-lockstep = { workspace = true } +serde = { workspace = true } serde_json = "1" hex = "0.4" diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/gossip.rs b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/gossip.rs deleted file mode 100644 index 145845339..000000000 --- a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/gossip.rs +++ /dev/null @@ -1,71 +0,0 @@ -//! What a node does with an object the gossip overlay hands it. -//! -//! Spec v1 describes the overlay in prose and defers its propagation rules to -//! a later revision (treatise §21.4, §21.6). What it does fix is enough to -//! decide every delivery the pack's gossip vectors make: -//! -//! * received bytes must be canonical, and are refused rather than re-encoded -//! when they are not (identity spec §2.5); -//! * they must hash to the SAID they state, and that SAID must be the one they -//! were announced or requested under (identity spec §2.1; treatise §10.2 -//! step 1, §21.4); -//! * a SAID already held is a duplicate: the overlay deduplicates by content -//! address (treatise §11.4). -//! -//! No first-come rule is needed. Two different canonical objects cannot share -//! a SAID short of a BLAKE3 collision, so an object that arrives under a held -//! SAID either is the held one or fails to hash to it. - -use smart_byte_core::{Said, said}; - -/// What became of one delivery. -#[derive(Clone, Debug, PartialEq, Eq)] -pub enum Delivery { - Accepted(Said), - Duplicate(Said), - /// Refused, with the pack's error class. - Rejected(&'static str), -} - -/// A node's store of objects received through the overlay. -#[derive(Debug, Default)] -pub struct Receiver { - held: Vec<(Said, Vec)>, -} - -impl Receiver { - pub fn new() -> Self { - Self::default() - } - - /// Deliver `bytes`, announced (or requested) under `claimed_said`. - pub fn deliver(&mut self, claimed_said: &str, bytes: &[u8]) -> Delivery { - let said = match said::verify(bytes, "d") { - Ok(s) => s, - Err(e) => return Delivery::Rejected(e.class()), - }; - match Said::from_said_text(claimed_said) { - Err(_) => return Delivery::Rejected("malformed_said"), - Ok(claimed) if claimed != said => return Delivery::Rejected("said_mismatch"), - Ok(_) => {} - } - if self.held.iter().any(|(s, _)| *s == said) { - return Delivery::Duplicate(said); - } - self.held.push((said, bytes.to_vec())); - Delivery::Accepted(said) - } - - /// The SAIDs held, in the order they were accepted. - pub fn held(&self) -> Vec { - self.held.iter().map(|(s, _)| *s).collect() - } - - /// The bytes held under `said`: exactly the bytes that were delivered. - pub fn bytes(&self, said: &Said) -> Option<&[u8]> { - self.held - .iter() - .find(|(s, _)| s == said) - .map(|(_, b)| b.as_slice()) - } -} diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/json.rs b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/json.rs index d25f0c5db..b6774524e 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/json.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/json.rs @@ -2,11 +2,15 @@ //! printer, and the typed mapping between JSON and the CBOR data model. //! //! JSON has no byte-string type, so the pack writes a byte string as an object -//! whose only member is `$bytes`, holding lowercase hex. Every other JSON value -//! maps to CBOR the obvious way: object → map with text keys, array → array, +//! whose only member is `$bytes`, holding lowercase hex. `$bytes` is reserved: +//! no map in the pack has a member of that name. Every other JSON value maps +//! to CBOR the obvious way: object → map with text keys, array → array, //! string → text, integer → integer, `true`/`false`/`null` → simple values. -//! A JSON number with a fraction or exponent is refused: floats have no -//! canonical form. +//! A JSON number with a fraction or exponent is refused: identity spec §2.3 +//! forbids floats. +//! +//! [`parse_strict`] reads the pack's JSON refusing duplicate object members, +//! which a plain JSON parser would silently collapse. use smart_byte_core::canonical::Value; @@ -137,8 +141,8 @@ pub fn bytes_json(b: &[u8]) -> J { /// the value holds them. /// /// # Panics -/// On a map key that is not text: the typed JSON has no spelling for one, and -/// the pack's generator never builds one. +/// On a map key that is not text, or a text key spelled `$bytes`: the typed +/// JSON has no spelling for either, and the pack's generator never builds one. pub fn to_json(v: &Value) -> J { match v { Value::Unsigned(n) => J::Int(i128::from(*n)), @@ -150,6 +154,7 @@ pub fn to_json(v: &Value) -> J { entries .iter() .map(|(k, v)| match k { + Value::Text(t) if t == "$bytes" => panic!("`$bytes` is a reserved key"), Value::Text(t) => (t.clone(), to_json(v)), other => panic!("typed JSON has text keys only, not {other:?}"), }) @@ -160,6 +165,79 @@ pub fn to_json(v: &Value) -> J { } } +/// Parse JSON, refusing any object that names a member twice. +pub fn parse_strict(bytes: &[u8]) -> Result { + use serde::de::DeserializeSeed; + let mut de = serde_json::Deserializer::from_slice(bytes); + let value = Strict.deserialize(&mut de).map_err(|e| e.to_string())?; + de.end().map_err(|e| e.to_string())?; + Ok(value) +} + +struct Strict; + +impl<'de> serde::de::DeserializeSeed<'de> for Strict { + type Value = serde_json::Value; + + fn deserialize>(self, d: D) -> Result { + d.deserialize_any(Strict) + } +} + +impl<'de> serde::de::Visitor<'de> for Strict { + type Value = serde_json::Value; + + fn expecting(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result { + f.write_str("JSON with no duplicate object members") + } + + fn visit_bool(self, b: bool) -> Result { + Ok(b.into()) + } + + fn visit_i64(self, n: i64) -> Result { + Ok(n.into()) + } + + fn visit_u64(self, n: u64) -> Result { + Ok(n.into()) + } + + fn visit_f64(self, n: f64) -> Result { + Ok(n.into()) + } + + fn visit_str(self, s: &str) -> Result { + Ok(s.into()) + } + + fn visit_unit(self) -> Result { + Ok(serde_json::Value::Null) + } + + fn visit_seq>(self, mut seq: A) -> Result { + let mut items = Vec::new(); + while let Some(item) = seq.next_element_seed(Strict)? { + items.push(item); + } + Ok(serde_json::Value::Array(items)) + } + + fn visit_map>(self, mut map: A) -> Result { + let mut out = serde_json::Map::new(); + while let Some(key) = map.next_key::()? { + if out.contains_key(&key) { + return Err(serde::de::Error::custom(format!( + "duplicate member {key:?}" + ))); + } + let value = map.next_value_seed(Strict)?; + out.insert(key, value); + } + Ok(serde_json::Value::Object(out)) + } +} + /// The CBOR value a typed-JSON value denotes. pub fn from_json(j: &serde_json::Value) -> Result { use serde_json::Value as S; @@ -172,17 +250,16 @@ pub fn from_json(j: &serde_json::Value) -> Result { } else if let Some(i) = n.as_i64() { Value::int(i) } else { - return Err(format!( - "{n} is not an integer; floats have no canonical form" - )); + return Err(format!("{n} is not an integer; floats are forbidden")); } } S::String(s) => Value::Text(s.clone()), S::Array(items) => Value::Array(items.iter().map(from_json).collect::>()?), S::Object(map) => { - if map.len() == 1 - && let Some(S::String(h)) = map.get("$bytes") - { + if let Some(h) = map.get("$bytes") { + let (1, S::String(h)) = (map.len(), h) else { + return Err("`$bytes` is reserved: a lone member holding hex".into()); + }; if h.len() % 2 != 0 || h.bytes().any(|c| !matches!(c, b'0'..=b'9' | b'a'..=b'f')) { return Err(format!( "$bytes must be lowercase hex of even length: {h:?}" @@ -226,15 +303,22 @@ mod tests { } #[test] - fn floats_and_loose_hex_are_refused() { + fn floats_loose_hex_and_misused_bytes_keys_are_refused() { assert!(from_json(&serde_json::json!(1.5)).is_err()); assert!(from_json(&serde_json::json!({"$bytes": "ABCD"})).is_err()); assert!(from_json(&serde_json::json!({"$bytes": "abc"})).is_err()); - // An object with `$bytes` and another member is a map, not bytes. - assert!(matches!( - from_json(&serde_json::json!({"$bytes": "00", "x": 1})).unwrap(), - Value::Map(_) - )); + // `$bytes` is reserved: never a map member alongside others, never + // anything but hex. + assert!(from_json(&serde_json::json!({"$bytes": "00", "x": 1})).is_err()); + assert!(from_json(&serde_json::json!({"$bytes": 7})).is_err()); + } + + #[test] + fn duplicate_members_are_refused() { + assert!(parse_strict(br#"{"a": 1, "b": {"c": 2}}"#).is_ok()); + assert!(parse_strict(br#"{"a": 1, "a": 2}"#).is_err()); + assert!(parse_strict(br#"[{"x": {"y": 1, "y": 1}}]"#).is_err()); + assert!(parse_strict(br#"{"a": 1} 2"#).is_err()); } #[test] diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/lib.rs b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/lib.rs index a03e8cdf0..217093d47 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/lib.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/lib.rs @@ -7,7 +7,6 @@ //! * `tests/pack.rs` runs both against the published pack under `cargo test`, //! so CI fails if the pack and the implementation ever disagree. -pub mod gossip; pub mod json; pub mod pack; pub mod verify; diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/pack.rs b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/pack.rs index ac3f76956..6615329d2 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/pack.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/pack.rs @@ -1,7 +1,8 @@ //! The v1 vector pack, built from the reference implementation. //! //! Inputs, names and descriptions are written here; every expected value is -//! computed by `smart-byte-core` and `smart-byte-lockstep` at generation time. +//! computed by `smart-byte-core` (canonical CBOR, SAIDs, signatures, the +//! inbox) and `smart-byte-lockstep` (the commit rule) at generation time. //! A negative vector also states the error class it is meant to produce, and //! generation fails if the reference refuses it for any other reason — so a //! vector cannot pass by accident while testing nothing. @@ -10,11 +11,11 @@ use std::collections::BTreeMap; use smart_byte_core::Said; use smart_byte_core::canonical::{self, Value}; -use smart_byte_core::said::{self, SAID_PLACEHOLDER, SaidDerivation}; +use smart_byte_core::inbox::{Delivery, Inbox}; +use smart_byte_core::said::{self, SAID_FIELD, SAID_PLACEHOLDER, SaidDerivation}; use smart_byte_core::sign::{self, SigningKey}; -use smart_byte_lockstep::{supermajority_threshold, tally}; +use smart_byte_lockstep::{node_view, supermajority_threshold, tally}; -use crate::gossip::{Delivery, Receiver}; use crate::json::{J, pretty, to_json}; /// The pack's vector files, in the order the README lists them. @@ -124,7 +125,7 @@ fn canonicalisation() -> J { ), canon_negative( "canon-004-floats-forbidden", - "Floating-point values have no canonical form: identity spec §2.3 forbids them, and a value that would be a float elsewhere is carried as a fixed-point integer. The input is a map holding 1.5 as a half-precision float, and a decoder refuses it.", + "Floats are forbidden (identity spec §2.3): a value that would be a float elsewhere is carried as a fixed-point integer pair. The input is a map holding 1.5 as a half-precision float, and a decoder refuses it.", &concat(&[&[0xa1], &cbor(&text("exact_half")), &[0xf9, 0x3e, 0x00]]), "float_forbidden", ), @@ -244,6 +245,12 @@ fn canonicalisation() -> J { &[0x62, 0xc3, 0x28], "invalid_utf8", ), + canon_negative( + "canon-020-simple-value-refused", + "The simple value undefined (0xf7). Identity spec §2.3 admits false, true and null and no other simple value, so the data model has nothing to decode it to and the input is refused.", + &[0xf7], + "simple_value_forbidden", + ), ]) } @@ -258,6 +265,8 @@ pub struct SaidVectors { pub by_name: BTreeMap<&'static str, SaidDerivation>, /// said-009's input: said-008's object with `n` written non-canonically. pub non_canonical_copy: Vec, + /// said-011's input: said-008's object with `n` changed, `d` kept. + pub changed_copy: Vec, } fn said_positive( @@ -266,16 +275,16 @@ fn said_positive( description: &str, input: Value, ) -> J { - let d = said::derive(&input, "d").expect(name); + let d = said::derive(&input, SAID_FIELD).expect(name); assert_eq!( - said::verify(&d.encoded, "d").as_ref(), + said::verify(&d.encoded, SAID_FIELD).as_ref(), Ok(&d.said), "{name}" ); let json = J::obj([ ("name", J::from(name)), ("description", J::from(description)), - ("said_field", J::from("d")), + ("said_field", J::from(SAID_FIELD)), ("input", to_json(&input)), ("canonical_cbor_hex", J::Str(hex::encode(&d.hashed))), ("blake3_hex", J::Str(hex::encode(d.said.as_bytes()))), @@ -293,7 +302,7 @@ fn said_negative( class: &str, extra: Vec<(&str, J)>, ) -> J { - let refused = said::verify(input, "d").expect_err(name); + let refused = said::verify(input, SAID_FIELD).expect_err(name); assert_eq!( refused.class(), class, @@ -302,7 +311,7 @@ fn said_negative( let mut entries = vec![ ("name", J::from(name)), ("description", J::from(description)), - ("said_field", J::from("d")), + ("said_field", J::from(SAID_FIELD)), ("input_cbor_hex", J::Str(hex::encode(input))), ("error", J::from(class)), ]; @@ -347,19 +356,19 @@ impl SaidVectors { said_positive( &mut by_name, "said-001-empty-envelope", - "The smallest envelope-shaped object (treatise §8.1): no provenance, no cargo, no ownership transitions, zero joule cost. It exercises the whole pipeline — placeholder, canonical CBOR, BLAKE3-256, `E` + base32 — on an object with no application data.", + "A nested object with an envelope's top-level members — provenance, ownership chain, cargo, joule cost — all empty or zero. It exercises the whole pipeline (placeholder, canonical CBOR, BLAKE3-256, `E` + base32) on an object with no application data.", envelope(Value::Null, vec![], Value::Null, joules(0, 0)), ), said_positive( &mut by_name, "said-002-usd-cargo", - "An envelope-shaped object carrying a USD cargo of 100.00 (100000000 micros) from issuer key 0x11…11. Cargo is opaque to the substrate; here it is a map whose members include a byte-string key.", + "The same object carrying a cargo map: a USD amount of 100.00 (100000000 micros) and a 32-byte issuer key. Members of mixed types — text, integer, byte string — all sort by their encoded keys.", envelope(Value::Null, vec![], usd.clone(), joules(12, 15)), ), said_positive( &mut by_name, "said-003-custom-cargo", - "The same shape carrying a non-money cargo: a temperature reading. The pipeline treats it exactly as it treats said-002 — cargo opacity is the property under test.", + "The same shape carrying a temperature reading as cargo. What a member means never affects its encoding: this cargo goes through the pipeline exactly as said-002's does.", envelope( Value::Null, vec![], @@ -375,7 +384,7 @@ impl SaidVectors { said_positive( &mut by_name, "said-004-with-provenance", - "An envelope-shaped object with a provenance attestation: algorithm identifier 0x01 (Ed25519), issuer key, issuance frame and signature. The SAID covers the provenance. The key and signature are fixed filler bytes: this vector tests SAID derivation, not signature validity, which signature_vectors.json covers.", + "The same shape with a provenance map: algorithm identifier 0x01 (Ed25519), issuer key, issuance frame and signature. The SAID covers every member. The key and signature are fixed filler bytes: this vector tests SAID derivation over them, not their validity.", envelope(provenance.clone(), vec![], usd.clone(), joules(14, 18)), ), ]; @@ -399,7 +408,7 @@ impl SaidVectors { vectors.push(said_positive( &mut by_name, "said-005-with-ownership-transitions", - "said-004's byte after two transfers, as fetched at frame 1100. The first transition's previous_history_hash is the digest of said-004's SAID, because a byte's identity is its initial history hash (treatise §10.1); the second is blake3 of that digest followed by the first transition's canonical encoding, illustrating §10.1's chain step (spec v1 does not pin the payload's serialization). This object's SAID differs from said-004's: a SAID is a function of the whole object, which is why a byte's identity is fixed at its origin (treatise §8.2) rather than recomputed as transitions accrue.", + "said-004's object with two ownership transitions appended and a larger joule cost: an array of maps holding byte strings. The first transition's previous_history_hash is the digest of said-004's SAID; the second is blake3 of that digest followed by the first transition's canonical encoding. The SAID differs from said-004's because a SAID covers the whole object it is derived over. A byte's identity is a different quantity — the hash of its origin attestation (treatise §8.2) — and does not change as transitions accrue.", envelope(provenance, vec![first, second], usd, joules(47, 56)), )); @@ -417,7 +426,7 @@ impl SaidVectors { vectors.push(said_positive( &mut by_name, "said-007-one-field-per-key", - "The one-field case: identity spec §3.4's per-key object {d, k}, whose SAID is a next-key commitment. k is the algorithm identifier 0x01 (Ed25519) followed by the public key of RFC 8032's first test seed, carried as a byte string.", + "The one-field case: identity spec §3.4's per-key object {d, k}, whose SAID is a next-key commitment. k is a byte string: the algorithm identifier 0x01 (Ed25519) followed by the public key of RFC 8032's first test seed (§3.3).", Value::map([("d", placeholder()), ("k", Value::Bytes(prefixed_key.clone()))]), )); assert_eq!( @@ -479,12 +488,12 @@ impl SaidVectors { ], )); - let mut changed = example.encoded.clone(); - *changed.last_mut().unwrap() = 0x02; + let mut changed_copy = example.encoded.clone(); + *changed_copy.last_mut().unwrap() = 0x02; vectors.push(said_negative( "said-011-changed-content-rejected", "said-008's object with n changed from 1 to 2 and its SAID left as it was. The bytes are canonical and well-formed, but they state a SAID they do not hash to.", - &changed, + &changed_copy, "said_mismatch", vec![], )); @@ -493,6 +502,7 @@ impl SaidVectors { json: J::Arr(vectors), by_name, non_canonical_copy, + changed_copy, } } @@ -510,14 +520,19 @@ fn signature_vector( name: &str, description: &str, said: &Said, - signing_seed: [u8; 32], + signing_seed: Option<[u8; 32]>, + verifying_key: [u8; 32], signature: &[u8], expected_verify: bool, error: Option<&str>, ) -> J { - let verifying_key = SigningKey::from_bytes(&signing_seed) - .verifying_key() - .to_bytes(); + if let Some(seed) = signing_seed { + assert_eq!( + SigningKey::from_bytes(&seed).verifying_key().to_bytes(), + verifying_key, + "{name}: a seed's vector carries the key the seed derives" + ); + } let result = sign::verify_said_signature(&said.to_said_text(), signature, &verifying_key); assert_eq!(result.is_ok(), expected_verify, "{name}"); assert_eq!( @@ -530,7 +545,12 @@ fn signature_vector( ("description", J::from(description)), ("envelope_said", J::Str(said.to_said_text())), ("signed_message_hex", J::Str(hex::encode(said.as_bytes()))), - ("signing_key_seed", J::Str(hex::encode(signing_seed))), + ( + "signing_key_seed", + signing_seed + .map(|s| J::Str(hex::encode(s))) + .unwrap_or(J::Null), + ), ("verifying_key", J::Str(hex::encode(verifying_key))), ("signature", J::Str(hex::encode(signature))), ("expected_verify", J::from(expected_verify)), @@ -558,6 +578,17 @@ fn with_s_plus_l(signature: &[u8; 64]) -> [u8; 64] { out } +/// The point y = 3 (sign bit clear) spelled with y = p + 3: an out-of-range +/// encoding of a valid, large-order point. +const KEY_Y_PLUS_P: [u8; 32] = [ + 0xf0, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x7f, +]; + +fn key_of(seed: &[u8; 32]) -> [u8; 32] { + SigningKey::from_bytes(seed).verifying_key().to_bytes() +} + fn signatures(saids: &SaidVectors) -> J { let empty = saids.said("said-001-empty-envelope").said; let usd = saids.said("said-002-usd-cargo").said; @@ -572,9 +603,10 @@ fn signatures(saids: &SaidVectors) -> J { J::Arr(vec![ signature_vector( "sig-001-known-key-known-said", - "Sign said-002's SAID with the key of RFC 8032's first test seed. The message is the 32 digest bytes the SAID spells (signed_message_hex), not its 53-character text. Ed25519 is deterministic, so every implementation produces exactly this signature.", + "Ed25519 (RFC 8032) is deterministic: signing said-002's SAID with the key of RFC 8032's first test seed gives exactly this signature. The message is the 32 digest bytes the SAID spells (signed_message_hex), not its 53-character text.", &usd, - seed1, + Some(seed1), + key_of(&seed1), &sig1, true, None, @@ -583,25 +615,28 @@ fn signatures(saids: &SaidVectors) -> J { "sig-002-verify-positive", "A signature over said-001's SAID by the key of RFC 8032's second test seed. It verifies under that key.", &empty, - seed2, + Some(seed2), + key_of(&seed2), &sig2, true, None, ), signature_vector( "sig-003-tampered-envelope", - "sig-002's signature presented with said-003's SAID: an adversary swaps the envelope and forwards the original signature. The signature is well-formed but binds another SAID, so verification MUST fail.", + "sig-002's signature presented with said-003's SAID: an adversary swaps the object and forwards the original signature. The signature is well-formed but binds another SAID, so verification MUST fail.", &custom, - seed2, + Some(seed2), + key_of(&seed2), &sig2, false, Some("signature_invalid"), ), signature_vector( "sig-004-wrong-key", - "sig-002's signature over said-001's SAID, presented with another key (the one the all-zero seed derives). The signature is valid for the key that made it, but that is not this key, so verification MUST fail: a valid signature from a key that is not the byte's owner.", + "sig-002's signature over said-001's SAID, presented with another key — the one the all-zero seed derives. The signature is valid for the key that made it, which is not this one, so verification MUST fail.", &empty, - zero, + Some(zero), + key_of(&zero), &sig2, false, Some("signature_invalid"), @@ -610,20 +645,32 @@ fn signatures(saids: &SaidVectors) -> J { "sig-005-malformed-signature", "sig-002's signature cut to 63 bytes. It is refused as malformed before any cryptography runs, and an implementation MUST NOT panic on it.", &empty, - seed2, + Some(seed2), + key_of(&seed2), &sig2[..63], false, Some("malformed_signature"), ), signature_vector( "sig-006-non-canonical-s-rejected", - "sig-002's signature with its scalar S replaced by S + L: the same scalar, written outside [0, L). RFC 8032 §5.1.7 requires refusing it. Accepting it would make signatures malleable, giving a second valid-looking signature for one message and key.", + "sig-002's signature with its scalar S replaced by S + L: the same scalar, written outside [0, L). RFC 8032 §5.1.7 requires refusing it; accepting it would make signatures malleable, giving a second valid-looking signature for one message and key.", &empty, - seed2, + Some(seed2), + key_of(&seed2), &with_s_plus_l(&sig2), false, Some("signature_invalid"), ), + signature_vector( + "sig-007-non-canonical-key-rejected", + "A verifying key spelled with y = p + 3: an out-of-range encoding of the valid, large-order point with y = 3. RFC 8032 §5.1.3 requires decoding to fail when y is not below p, so the key is refused as malformed before any verification runs — one key has one spelling. There is no seed: no key pair has this spelling.", + &empty, + None, + KEY_Y_PLUS_P, + &sig2, + false, + Some("malformed_verifying_key"), + ), ]) } @@ -631,85 +678,78 @@ fn signatures(saids: &SaidVectors) -> J { // lockstep_vectors.json // --------------------------------------------------------------------------- +/// One node: its name, the hash it reported, and the hash it computed. +type NodeHashes<'a> = (&'a str, u8, u8); + fn lockstep_vector( name: &str, description: &str, frame: u64, cluster_size: usize, - reports: &[(&str, u8)], + nodes: &[NodeHashes<'_>], ) -> J { - let reports: Vec<(String, [u8; 32])> = reports + let reported: Vec<(String, [u8; 32])> = nodes .iter() - .map(|(node, fill)| ((*node).to_owned(), [*fill; 32])) + .map(|(node, r, _)| ((*node).to_owned(), [*r; 32])) .collect(); - let t = tally(cluster_size, &reports).expect(name); + // What the broadcast hashes decide: the frame's public record. + let public = tally(cluster_size, &reported).expect(name); - let mut entries = Vec::new(); - for (_, hash) in &reports { - if let Some(entry) = t.dissent_entry(frame, hash) - && !entries.contains(&entry) - { - entries.push(entry); - } + let mut outcomes = Vec::new(); + let mut logs = Vec::new(); + for (node, _, c) in nodes { + let computed = [*c; 32]; + let view = node_view(cluster_size, &reported, &(*node).to_owned(), &computed).expect(name); + outcomes.push(( + (*node).to_owned(), + J::from(view.outcome(&computed).as_str()), + )); + let log: Vec = view + .dissent_entry(frame, &computed) + .into_iter() + .map(|e| { + J::obj([ + ("frame", J::from(e.frame)), + ("dissenting_peers", J::strs(e.dissenting_peers)), + ("local_hash", J::Str(hex::encode(e.local_hash))), + ( + "supermajority_hash", + J::Str(hex::encode(e.supermajority_hash)), + ), + ]) + }) + .collect(); + logs.push(((*node).to_owned(), J::Arr(log))); } - // §14.2: every committing node writes the same entry, so there is at - // most one per frame. - assert!( - entries.len() <= 1, - "{name}: committing nodes disagree on the dissent entry" - ); + let hashes = |pick: fn(&NodeHashes<'_>) -> u8| { + J::obj( + nodes + .iter() + .map(|n| (n.0.to_owned(), J::Str(hex::encode([pick(n); 32])))), + ) + }; J::obj([ ("name", J::from(name)), ("description", J::from(description)), ("node_count", J::from(cluster_size)), ("frame_number", J::from(frame)), - ( - "node_hashes", - J::obj( - reports - .iter() - .map(|(n, h)| (n.clone(), J::Str(hex::encode(h)))), - ), - ), + ("node_reported_hashes", hashes(|n| n.1)), + ("node_computed_hashes", hashes(|n| n.2)), ( "expected_threshold", J::from(supermajority_threshold(cluster_size)), ), - ("expected_commit", J::from(t.committed())), + ("expected_commit", J::from(public.committed())), ( "expected_state_hash", - t.supermajority_hash + public + .supermajority_hash .map(|h| J::Str(hex::encode(h))) .unwrap_or(J::Null), ), - ( - "expected_outcomes", - J::obj( - reports - .iter() - .map(|(n, h)| (n.clone(), J::from(t.outcome(h).as_str()))), - ), - ), - ( - "expected_dissent", - J::Arr( - entries - .into_iter() - .map(|e| { - J::obj([ - ("frame", J::from(e.frame)), - ("dissenting_peers", J::strs(e.dissenting_peers)), - ("local_hash", J::Str(hex::encode(e.local_hash))), - ( - "supermajority_hash", - J::Str(hex::encode(e.supermajority_hash)), - ), - ]) - }) - .collect(), - ), - ), + ("expected_outcomes", J::Obj(outcomes)), + ("expected_dissent_logs", J::Obj(logs)), ]) } @@ -717,60 +757,87 @@ fn lockstep() -> J { J::Arr(vec![ lockstep_vector( "lockstep-001-unanimous-commit", - "All four nodes report the same post-frame state hash. The threshold is floor(2·4/3)+1 = 3, so the frame commits, every node's outcome is a clean commit (treatise §13.3), and nothing is logged as dissent. State hashes in these vectors are opaque 32-byte values compared only for equality: §9.4 does not pin the state's serialization, so the commit rule is tested over the hashes nodes report.", + "All four nodes report the post-frame state hash they computed, and all agree. The threshold is floor(2·4/3)+1 = 3, so the frame commits, every node's outcome is a clean commit (treatise §13.3), and no dissent is logged. State hashes are opaque 32-byte values compared for equality. Each node's outcome comes from its own view: the hashes its peers reported plus the one it computed itself (§13.3, §13.4).", 184237, 4, &[ - ("node_1", 0xaa), - ("node_2", 0xaa), - ("node_3", 0xaa), - ("node_4", 0xaa), + ("node_1", 0xaa, 0xaa), + ("node_2", 0xaa, 0xaa), + ("node_3", 0xaa, 0xaa), + ("node_4", 0xaa, 0xaa), ], ), lockstep_vector( "lockstep-002-supermajority-commit", - "Three of four nodes agree and node_4 reports another hash. The threshold of 3 is met and the frame commits at the majority hash. Nodes 1–3 commit with dissent and write one dissent-log entry naming node_4 (treatise §14.1). node_4's own outcome is minority divergence: it halts, and writes nothing.", + "Three of four nodes agree. node_4's state diverged, and it reports the hash it computed. The threshold of 3 is met: nodes 1–3 commit with dissent, and each writes one dissent-log entry naming node_4 (treatise §14.1). node_4 sees a supermajority on a hash that is not its own — minority divergence — so it halts and writes nothing.", 184238, 4, &[ - ("node_1", 0xbb), - ("node_2", 0xbb), - ("node_3", 0xbb), - ("node_4", 0xdd), + ("node_1", 0xbb, 0xbb), + ("node_2", 0xbb, 0xbb), + ("node_3", 0xbb, 0xbb), + ("node_4", 0xdd, 0xdd), ], ), lockstep_vector( "lockstep-003-threshold-not-met", - "No hash is reported by more than two of the four nodes. None reaches the threshold of 3, so the frame does not commit and every node's outcome is a cluster split (§13.3). No dissent is recorded: dissent is divergence from a committed hash (§14.2), and there is none.", + "No hash is reported by more than two of the four nodes. None reaches the threshold of 3, so the frame does not commit and every node's outcome is a cluster split (§13.3). No dissent is logged: dissent is divergence from a committed hash (§14.2), and there is none.", 184239, 4, &[ - ("node_1", 0xcc), - ("node_2", 0xcc), - ("node_3", 0xdd), - ("node_4", 0xee), + ("node_1", 0xcc, 0xcc), + ("node_2", 0xcc, 0xcc), + ("node_3", 0xdd, 0xdd), + ("node_4", 0xee, 0xee), ], ), lockstep_vector( - "lockstep-004-three-nodes-one-liar-split", - "The n = 3 exception (treatise §13.2). The threshold is 3, so one node reporting a different hash leaves 2 of the 3 votes the frame needs. The cluster detects the divergence but cannot tolerate it: no commit, and every node halts with a cluster split.", + "lockstep-004-three-nodes-one-diverged-split", + "The n = 3 exception (treatise §13.2). The threshold is 3, so one node whose state diverged leaves 2 of the 3 votes the frame needs. The cluster detects the divergence but cannot tolerate it: every node halts with a cluster split.", 184240, 3, - &[("node_1", 0x11), ("node_2", 0x11), ("node_3", 0x22)], + &[ + ("node_1", 0x11, 0x11), + ("node_2", 0x11, 0x11), + ("node_3", 0x22, 0x22), + ], ), lockstep_vector( "lockstep-005-seven-nodes-two-dissenters", - "Seven nodes, threshold 5: five agree and two report hashes of their own. The frame commits. The dissent-log entry names both dissenters in ascending order, so every honest node writes the same entry (§14.2).", + "Seven nodes, threshold 5: five agree and two diverged nodes report hashes of their own. The frame commits. The dissent entry names both dissenters in ascending order, so every committing node writes the same entry (§14.2).", 184241, 7, &[ - ("node_1", 0x5a), - ("node_2", 0x5a), - ("node_3", 0x5a), - ("node_4", 0x5a), - ("node_5", 0x5a), - ("node_6", 0x6b), - ("node_7", 0x7c), + ("node_1", 0x5a, 0x5a), + ("node_2", 0x5a, 0x5a), + ("node_3", 0x5a, 0x5a), + ("node_4", 0x5a, 0x5a), + ("node_5", 0x5a, 0x5a), + ("node_6", 0x6b, 0x6b), + ("node_7", 0x7c, 0x7c), + ], + ), + lockstep_vector( + "lockstep-006-transport-liar-is-logged", + "Treatise §13.4: node_4 computed the honest hash but broadcast another. Its peers see 3-of-4 agreement and record node_4 in their dissent logs. node_4 tallies its own honest hash, sees unanimity and commits cleanly: its lie surfaces only in its peers' logs, which is why the dissent log is the substrate's misbehaviour signal (§14).", + 184242, + 4, + &[ + ("node_1", 0xbb, 0xbb), + ("node_2", 0xbb, 0xbb), + ("node_3", 0xbb, 0xbb), + ("node_4", 0xdd, 0xbb), + ], + ), + lockstep_vector( + "lockstep-007-three-node-transport-liar", + "§13.2 and §13.4 together. At n = 3 a node that broadcasts a hash other than the one it computed splits the honest pair, who halt. The liar sees its own honest hash agree with both peers and commits. The reported hashes reach no supermajority, so the frame's public record is no commit.", + 184243, + 3, + &[ + ("node_1", 0x11, 0x11), + ("node_2", 0x11, 0x11), + ("node_3", 0x22, 0x11), ], ), ]) @@ -786,18 +853,10 @@ fn gossip_vector( receiver: &str, deliveries: &[(&str, String, Vec)], ) -> J { - let mut node = Receiver::new(); + let mut inbox = Inbox::new(); let mut results = Vec::new(); for (_, claimed, bytes) in deliveries { - let result = node.deliver(claimed, bytes); - if let Delivery::Accepted(s) = &result { - assert_eq!( - node.bytes(s), - Some(bytes.as_slice()), - "{name}: held bytes must be the delivered bytes" - ); - } - results.push(match result { + results.push(match inbox.deliver(claimed, bytes) { Delivery::Accepted(s) => J::obj([ ("outcome", J::from("accepted")), ("said", J::Str(s.to_said_text())), @@ -806,9 +865,10 @@ fn gossip_vector( ("outcome", J::from("duplicate")), ("said", J::Str(s.to_said_text())), ]), - Delivery::Rejected(class) => { - J::obj([("outcome", J::from("rejected")), ("error", J::from(class))]) - } + Delivery::Rejected(e) => J::obj([ + ("outcome", J::from("rejected")), + ("error", J::from(e.class())), + ]), }); } J::obj([ @@ -833,74 +893,74 @@ fn gossip_vector( ("expected_results", J::Arr(results)), ( "expected_held", - J::strs(node.held().iter().map(Said::to_said_text)), + J::Arr( + inbox + .held() + .iter() + .map(|(said, bytes)| { + J::obj([ + ("said", J::Str(said.to_said_text())), + ("cbor_hex", J::Str(hex::encode(bytes))), + ]) + }) + .collect(), + ), ), ]) } fn gossip(saids: &SaidVectors) -> J { - let usd = saids.said("said-002-usd-cargo"); - let custom = saids.said("said-003-custom-cargo"); - let origin = saids.said("said-004-with-provenance"); - let transferred = saids.said("said-005-with-ownership-transitions"); + let empty = saids.said("said-006-only-said-field"); + let per_key = saids.said("said-007-one-field-per-key"); let example = saids.said("said-008-spec-worked-example"); - - // said-003's object with its reading changed and its SAID left as it was. - let mut altered = custom.object.clone(); - *altered - .get_mut("cargo") - .and_then(|c| c.get_mut("celsius_milli")) - .unwrap() = uint(99999); - let altered = cbor(&altered); - let text = |d: &SaidDerivation| d.said.to_said_text(); J::Arr(vec![ gossip_vector( "gossip-001-cross-node-propagation", - "An envelope published at node_A reaches node_B. node_B accepts it because its bytes are canonical and hash to the SAID they state, which is the SAID it was announced under, and it holds the bytes unchanged. Gossip carries the object itself, never a re-encoding of it.", + "An object published at node_A reaches node_B. Its bytes are canonical, they hash to the SAID they state, and that is the SAID they were announced under, so node_B accepts it (identity spec §2.6) and holds exactly the bytes it received. Gossip carries the object itself, never a re-encoding of it.", "node_B", - &[("node_A", text(usd), usd.encoded.clone())], + &[("node_A", text(per_key), per_key.encoded.clone())], ), gossip_vector( "gossip-002-said-conflict-rejected", - "node_C holds said-003's object when a second object arrives under the same SAID: said-003's with its reading changed and its SAID left as it was. Content addressing settles the conflict without a first-come rule: the newcomer does not hash to the SAID it states, so it is refused. Two different canonical objects cannot share a SAID short of a BLAKE3 collision.", + "node_C holds said-008's object when a second object arrives under the same SAID: said-008's with n changed and its SAID left as it was (said-011's bytes). Verification comes before deduplication (identity spec §2.6), so the newcomer is reported as a SAID mismatch, which makes the substitution attempt visible, instead of being absorbed as a duplicate of what node_C holds.", "node_C", &[ - ("node_A", text(custom), custom.encoded.clone()), - ("node_B", text(custom), altered.clone()), + ("node_A", text(example), example.encoded.clone()), + ("node_B", text(example), saids.changed_copy.clone()), ], ), gossip_vector( "gossip-003-backfill-answer-verified", - "node_B lacks the object behind said-004's SAID and asks for it by identity: any party holding an identifier can request the rest from any peer (treatise §21.4). node_A answers with said-005's object, which is well-formed and self-consistent but not the one requested, so it is refused. node_D's answer hashes to the requested SAID and is accepted. Spec v1 leaves gossip propagation rules to a later revision (treatise §21.6); what it does fix is that an answer is checked against the identifier that was asked for.", + "node_B requests the object behind said-007's SAID: a party holding an identifier can request the object from any peer (treatise §21.4). node_A answers with said-008's object, which is genuine and self-consistent but not the one requested, and is refused. node_D's answer hashes to the requested SAID and is accepted (identity spec §2.6).", "node_B", &[ - ("node_A", text(origin), transferred.encoded.clone()), - ("node_D", text(origin), origin.encoded.clone()), + ("node_A", text(per_key), example.encoded.clone()), + ("node_D", text(per_key), per_key.encoded.clone()), ], ), gossip_vector( "gossip-004-duplicate-delivery-deduplicated", - "The same object reaches node_B from two peers. The second delivery is a duplicate of a SAID already held and changes nothing: the overlay deduplicates by content address (treatise §11.4).", + "The same object reaches node_B from two peers. The second delivery verifies and is a duplicate of a SAID already held, so it changes nothing: the overlay deduplicates by content address (treatise §11.4).", "node_B", &[ - ("node_A", text(usd), usd.encoded.clone()), - ("node_C", text(usd), usd.encoded.clone()), + ("node_A", text(empty), empty.encoded.clone()), + ("node_C", text(empty), empty.encoded.clone()), ], ), gossip_vector( "gossip-005-non-canonical-copy-rejected", - "said-009's bytes reach node_B announced under said-008's SAID, the SAID a re-encoding receiver would compute for them. The bytes are not canonical, so a conforming receiver refuses them (identity spec §2.5). Accepting them would let a second byte string circulate under said-008's SAID.", + "said-009's bytes reach node_B announced under said-008's SAID, the SAID a re-encoding receiver would compute for them. The bytes are not canonical, so node_B refuses them (identity spec §2.5, §2.6). Accepting them would let a second byte string circulate under said-008's SAID.", "node_B", &[("node_A", text(example), saids.non_canonical_copy.clone())], ), gossip_vector( "gossip-006-substitute-arriving-first-rejected", - "gossip-002 in the other order: the altered object reaches node_C first and is refused, then said-003's object arrives and is accepted. Arrival order decides nothing; the content does.", + "gossip-002 in the other order: the altered object reaches node_C first and is refused, then said-008's object arrives and is accepted. Arrival order decides nothing; the content does.", "node_C", &[ - ("node_B", text(custom), altered), - ("node_A", text(custom), custom.encoded.clone()), + ("node_B", text(example), saids.changed_copy.clone()), + ("node_A", text(example), example.encoded.clone()), ], ), ]) diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/verify.rs b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/verify.rs index 9ce7a2cf1..a852e105e 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/verify.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/src/verify.rs @@ -4,19 +4,24 @@ //! from the vector's own inputs, and every negative vector must be refused //! with the error class it names. It does not consult the generator, so it //! checks a pack whoever produced it. +//! +//! It also checks that the pack is complete, against texts that live outside +//! the pack: every vector the identity spec cites must be present, and every +//! error class the conformance README lists must be exercised (and no other). +//! A pack cannot pass by losing vectors. use std::collections::{BTreeMap, BTreeSet}; -use std::path::Path; +use std::path::{Path, PathBuf}; use serde_json::Value as S; use smart_byte_core::Said; use smart_byte_core::canonical::{self, Value}; +use smart_byte_core::inbox::{Delivery, Inbox}; use smart_byte_core::said::{self, SAID_PLACEHOLDER}; use smart_byte_core::sign::{self, SigningKey}; -use smart_byte_lockstep::{supermajority_threshold, tally}; +use smart_byte_lockstep::{node_view, supermajority_threshold, tally}; -use crate::gossip::{Delivery, Receiver}; -use crate::json::from_json; +use crate::json::{from_json, parse_strict}; use crate::pack::FILES; /// One vector's result. @@ -67,6 +72,13 @@ fn hex_field(v: &S, key: &str) -> Result, String> { hex::decode(text(v, key)?).map_err(|e| format!("`{key}` is not hex: {e}")) } +fn hash32(v: &S, what: &str) -> Result<[u8; 32], String> { + v.as_str() + .and_then(|s| hex::decode(s).ok()) + .and_then(|b| b.try_into().ok()) + .ok_or_else(|| format!("{what} is not 32 bytes of hex")) +} + fn same(what: &str, computed: impl PartialEq + std::fmt::Display, published: &str) -> R { if computed == published.to_owned() { Ok(()) @@ -77,6 +89,59 @@ fn same(what: &str, computed: impl PartialEq + std::fmt::Display, publis } } +/// The texts the completeness checks are held to: the identity spec and the +/// conformance README in this repository. +fn repo_text(relative: &str) -> Result { + let path = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join(relative); + std::fs::read_to_string(&path).map_err(|e| format!("read {}: {e}", path.display())) +} + +/// The SAID vector prefixes the identity spec cites (`said-006`, and every +/// member of a range written `said-001` to `said-005`). +pub fn spec_cited_said_vectors() -> Result, String> { + let spec = repo_text("../../../spec/identity_and_key_rotation.md")?; + let mut cited = BTreeSet::new(); + let words: Vec<&str> = spec + .split(|c: char| !(c.is_ascii_alphanumeric() || c == '-')) + .filter(|w| !w.is_empty()) + .collect(); + let number = |w: &str| w.strip_prefix("said-").and_then(|n| n.parse::().ok()); + for (i, w) in words.iter().enumerate() { + let Some(n) = number(w) else { continue }; + cited.insert(format!("said-{n:03}")); + if words.get(i + 1) == Some(&"to") + && let Some(end) = words.get(i + 2).and_then(|w| number(w)) + { + for k in n..=end { + cited.insert(format!("said-{k:03}")); + } + } + } + if cited.is_empty() { + return Err("the identity spec cites no SAID vector".into()); + } + Ok(cited) +} + +/// The error classes the conformance README's table lists. +pub fn readme_error_classes() -> Result, String> { + let readme = repo_text("../../../conformance/README.md")?; + let section = readme + .split("### Error classes") + .nth(1) + .ok_or("the README has no error-class table")?; + let classes: BTreeSet = section + .lines() + .skip_while(|l| !l.starts_with('|')) + .take_while(|l| l.starts_with('|')) + .filter_map(|l| l.split('`').nth(1).map(str::to_owned)) + .collect(); + if classes.is_empty() { + return Err("the README's error-class table is empty".into()); + } + Ok(classes) +} + /// Check every file of the pack in `dir`. pub fn verify_pack(dir: &Path) -> Report { let mut report = Report::default(); @@ -85,7 +150,7 @@ pub fn verify_pack(dir: &Path) -> Report { for file in FILES { let loaded = std::fs::read(dir.join(file)) .map_err(|e| format!("read: {e}")) - .and_then(|b| serde_json::from_slice::(&b).map_err(|e| format!("parse: {e}"))) + .and_then(|b| parse_strict(&b).map_err(|e| format!("parse: {e}"))) .and_then(|j| match j { S::Array(items) => Ok(items), _ => Err("the top-level value must be an array".to_owned()), @@ -124,11 +189,26 @@ pub fn verify_pack(dir: &Path) -> Report { report.record(file, name, result); } } + + report.record( + "(pack)", + "spec-cited vectors present", + cited_present(&files), + ); + report.record( + "(pack)", + "README error classes exercised", + classes_exercised(&files), + ); report } -/// Names and descriptions present, names unique, supersessions resolvable. +/// Names and descriptions present, names unique, supersessions resolvable, +/// and at least one vector. fn structure(items: &[S]) -> R { + if items.is_empty() { + return Err("the file holds no vectors".into()); + } let mut names = BTreeSet::new(); for v in items { let name = text(v, "name")?; @@ -150,6 +230,55 @@ fn structure(items: &[S]) -> R { Ok(()) } +fn cited_present(files: &BTreeMap<&str, Vec>) -> R { + let names: Vec<&str> = files + .get("said_vectors.json") + .into_iter() + .flatten() + .filter_map(|v| v.get("name").and_then(S::as_str)) + .collect(); + let missing: Vec = spec_cited_said_vectors()? + .into_iter() + .filter(|prefix| !names.iter().any(|n| n.starts_with(&format!("{prefix}-")))) + .collect(); + if missing.is_empty() { + Ok(()) + } else { + Err(format!( + "the identity spec cites vectors the pack lacks: {missing:?}" + )) + } +} + +fn classes_exercised(files: &BTreeMap<&str, Vec>) -> R { + let mut used = BTreeSet::new(); + for v in files.values().flatten() { + if let Some(e) = v.get("error").and_then(S::as_str) { + used.insert(e.to_owned()); + } + for r in v + .get("expected_results") + .and_then(S::as_array) + .into_iter() + .flatten() + { + if let Some(e) = r.get("error").and_then(S::as_str) { + used.insert(e.to_owned()); + } + } + } + let listed = readme_error_classes()?; + let unexercised: Vec<&String> = listed.difference(&used).collect(); + let undocumented: Vec<&String> = used.difference(&listed).collect(); + if unexercised.is_empty() && undocumented.is_empty() { + Ok(()) + } else { + Err(format!( + "README classes no vector exercises: {unexercised:?}; classes the README does not list: {undocumented:?}" + )) + } +} + fn refused_as(what: &str, class: Result<(), &'static str>, v: &S) -> R { let want = text(v, "error")?; match class { @@ -250,8 +379,13 @@ fn said_vector(v: &S) -> R { /// received bytes with the stated SAID overwritten in place by the placeholder. fn said_of_received_bytes(input: &[u8], field_name: &str) -> Result { let stated = match lenient_decode(input)?.get(field_name) { - Some(Value::Text(t)) => t.clone(), - _ => return Err(format!("no text `{field_name}` to substitute")), + Some(Value::Text(t)) if t.len() == SAID_PLACEHOLDER.len() => t.clone(), + _ => { + return Err(format!( + "no {}-character `{field_name}` to substitute", + SAID_PLACEHOLDER.len() + )); + } }; let at = input .windows(stated.len()) @@ -266,7 +400,10 @@ fn said_of_received_bytes(input: &[u8], field_name: &str) -> Result Result { - fn item(b: &[u8], pos: &mut usize) -> Result { + fn item(b: &[u8], pos: &mut usize, depth: usize) -> Result { + if depth > canonical::MAX_DEPTH { + return Err("too deep".into()); + } let initial = *b.get(*pos).ok_or("truncated")?; *pos += 1; let (major, ai) = (initial >> 5, initial & 0x1f); @@ -294,7 +431,8 @@ fn lenient_decode(input: &[u8]) -> Result { *pos += 1; } let mut take = |n: u64| -> Result, String> { - let end = *pos + usize::try_from(n).map_err(|e| e.to_string())?; + let n = usize::try_from(n).map_err(|e| e.to_string())?; + let end = pos.checked_add(n).ok_or("truncated")?; let s = b.get(*pos..end).ok_or("truncated")?.to_vec(); *pos = end; Ok(s) @@ -304,17 +442,26 @@ fn lenient_decode(input: &[u8]) -> Result { 1 => Value::Negative(arg), 2 => Value::Bytes(take(arg)?), 3 => Value::Text(String::from_utf8(take(arg)?).map_err(|e| e.to_string())?), - 4 => Value::Array((0..arg).map(|_| item(b, pos)).collect::>()?), - 5 => Value::Map( - (0..arg) - .map(|_| Ok((item(b, pos)?, item(b, pos)?))) - .collect::>()?, - ), + 4 => { + let mut items = Vec::new(); + for _ in 0..arg { + items.push(item(b, pos, depth + 1)?); + } + Value::Array(items) + } + 5 => { + let mut entries = Vec::new(); + for _ in 0..arg { + let k = item(b, pos, depth + 1)?; + entries.push((k, item(b, pos, depth + 1)?)); + } + Value::Map(entries) + } _ => return Err(format!("lenient decoder: unsupported major type {major}")), }) } let mut pos = 0; - let v = item(input, &mut pos)?; + let v = item(input, &mut pos, 0)?; if pos != input.len() { return Err("trailing bytes".into()); } @@ -334,15 +481,19 @@ fn signature(v: &S, said_texts: &BTreeSet) -> R { hex::encode(said.as_bytes()), text(v, "signed_message_hex")?, )?; - let seed: [u8; 32] = hex_field(v, "signing_key_seed")? - .try_into() - .map_err(|_| "signing_key_seed is not 32 bytes".to_owned())?; - let key = SigningKey::from_bytes(&seed); - same( - "verifying_key", - hex::encode(key.verifying_key().to_bytes()), - text(v, "verifying_key")?, - )?; + let key = match field(v, "signing_key_seed")? { + S::Null => None, + seed => { + let seed: [u8; 32] = hash32(seed, "signing_key_seed")?; + let key = SigningKey::from_bytes(&seed); + same( + "verifying_key", + hex::encode(key.verifying_key().to_bytes()), + text(v, "verifying_key")?, + )?; + Some(key) + } + }; let sig = hex_field(v, "signature")?; let result = sign::verify_said_signature(said_text, &sig, &hex_field(v, "verifying_key")?); match field(v, "expected_verify")?.as_bool() { @@ -351,7 +502,9 @@ fn signature(v: &S, said_texts: &BTreeSet) -> R { if !field(v, "error")?.is_null() { return Err("a positive vector names an error".into()); } - // Ed25519 is deterministic: the published signature is THE signature. + // Ed25519 is deterministic: the published signature is THE + // signature, so a positive vector must carry its seed. + let key = key.ok_or("a positive vector must carry its signing seed")?; same( "signature", hex::encode(sign::sign_said(&said, &key).to_bytes()), @@ -370,19 +523,20 @@ fn lockstep(v: &S) -> R { let frame = field(v, "frame_number")? .as_u64() .ok_or("frame_number is not an integer")?; - let hashes = field(v, "node_hashes")? + let reported_map = field(v, "node_reported_hashes")? .as_object() - .ok_or("node_hashes is not an object")?; - let mut reports = Vec::new(); - for (node, h) in hashes { - let h: [u8; 32] = h - .as_str() - .and_then(|s| hex::decode(s).ok()) - .and_then(|b| b.try_into().ok()) - .ok_or_else(|| format!("{node}'s hash is not 32 bytes of hex"))?; - reports.push((node.clone(), h)); + .ok_or("node_reported_hashes is not an object")?; + let computed_map = field(v, "node_computed_hashes")? + .as_object() + .ok_or("node_computed_hashes is not an object")?; + if reported_map.keys().ne(computed_map.keys()) { + return Err("reported and computed hashes must name the same nodes".into()); } - let t = tally(n, &reports).map_err(|e| e.to_string())?; + let mut reported = Vec::new(); + for (node, h) in reported_map { + reported.push((node.clone(), hash32(h, node)?)); + } + let public = tally(n, &reported).map_err(|e| e.to_string())?; same( "expected_threshold", @@ -391,50 +545,54 @@ fn lockstep(v: &S) -> R { )?; same( "expected_commit", - t.committed().to_string(), + public.committed().to_string(), &field(v, "expected_commit")?.to_string(), )?; - let state = t.supermajority_hash.map(hex::encode); + let state = public.supermajority_hash.map(hex::encode); if field(v, "expected_state_hash")?.as_str() != state.as_deref() { return Err(format!("expected_state_hash: reference computes {state:?}")); } + let outcomes = field(v, "expected_outcomes")? .as_object() .ok_or("expected_outcomes is not an object")?; - if outcomes.len() != reports.len() { - return Err("expected_outcomes must name every reporting node".into()); + let logs = field(v, "expected_dissent_logs")? + .as_object() + .ok_or("expected_dissent_logs is not an object")?; + if outcomes.len() != reported.len() || logs.len() != reported.len() { + return Err("expected_outcomes and expected_dissent_logs must name every node".into()); } - for (node, h) in &reports { + for (node, computed) in computed_map { + let computed = hash32(computed, node)?; + let view = node_view(n, &reported, node, &computed).map_err(|e| e.to_string())?; let published = outcomes .get(node) .and_then(S::as_str) .ok_or_else(|| format!("no outcome for {node}"))?; same( &format!("{node}'s outcome"), - t.outcome(h).as_str().to_owned(), + view.outcome(&computed).as_str().to_owned(), published, )?; - } - let mut entries: Vec = Vec::new(); - for (_, h) in &reports { - if let Some(e) = t.dissent_entry(frame, h) { - let j = serde_json::json!({ - "frame": e.frame, - "dissenting_peers": e.dissenting_peers, - "local_hash": hex::encode(e.local_hash), - "supermajority_hash": hex::encode(e.supermajority_hash), - }); - if !entries.contains(&j) { - entries.push(j); - } + let log: Vec = view + .dissent_entry(frame, &computed) + .into_iter() + .map(|e| { + serde_json::json!({ + "frame": e.frame, + "dissenting_peers": e.dissenting_peers, + "local_hash": hex::encode(e.local_hash), + "supermajority_hash": hex::encode(e.supermajority_hash), + }) + }) + .collect(); + if logs.get(node) != Some(&S::Array(log.clone())) { + return Err(format!( + "{node}'s dissent log: reference computes {}", + S::Array(log) + )); } } - if field(v, "expected_dissent")? != &S::Array(entries.clone()) { - return Err(format!( - "expected_dissent: reference computes {}", - S::Array(entries) - )); - } Ok(()) } @@ -448,16 +606,18 @@ fn gossip(v: &S) -> R { if deliveries.len() != expected.len() { return Err("one expected result per delivery".into()); } - let mut node = Receiver::new(); + let mut inbox = Inbox::new(); for (i, (d, want)) in deliveries.iter().zip(expected).enumerate() { - let got = match node.deliver(text(d, "claimed_said")?, &hex_field(d, "cbor_hex")?) { + let got = match inbox.deliver(text(d, "claimed_said")?, &hex_field(d, "cbor_hex")?) { Delivery::Accepted(s) => { serde_json::json!({"outcome": "accepted", "said": s.to_said_text()}) } Delivery::Duplicate(s) => { serde_json::json!({"outcome": "duplicate", "said": s.to_said_text()}) } - Delivery::Rejected(class) => serde_json::json!({"outcome": "rejected", "error": class}), + Delivery::Rejected(e) => { + serde_json::json!({"outcome": "rejected", "error": e.class()}) + } }; if &got != want { return Err(format!( @@ -465,9 +625,13 @@ fn gossip(v: &S) -> R { )); } } - let held: Vec = node.held().iter().map(Said::to_said_text).collect(); - if field(v, "expected_held")? != &serde_json::json!(held) { - return Err(format!("expected_held: reference holds {held:?}")); + let held: Vec = inbox + .held() + .iter() + .map(|(s, b)| serde_json::json!({"said": s.to_said_text(), "cbor_hex": hex::encode(b)})) + .collect(); + if field(v, "expected_held")? != &S::Array(held.clone()) { + return Err(format!("expected_held: reference holds {}", S::Array(held))); } Ok(()) } diff --git a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/tests/pack.rs b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/tests/pack.rs index 4b6b6547f..32a636390 100644 --- a/smart-byte/smart-byte-rs/crates/smart-byte-conformance/tests/pack.rs +++ b/smart-byte/smart-byte-rs/crates/smart-byte-conformance/tests/pack.rs @@ -5,6 +5,7 @@ use std::collections::BTreeSet; use std::path::Path; use serde_json::{Value, json}; +use smart_byte_conformance::verify::{readme_error_classes, spec_cited_said_vectors}; use smart_byte_conformance::{pack, published_pack_dir, verify::verify_pack}; fn published(file: &str) -> String { @@ -32,9 +33,10 @@ fn every_published_vector_agrees_with_the_reference() { let report = verify_pack(&published_pack_dir()); let failures = report.failures(); assert!(failures.is_empty(), "{failures:#?}"); - // One structure check per file and one check per vector: none skipped. + // One structure check per file, one check per vector, two completeness + // checks: none skipped. let total: usize = pack::FILES.iter().map(|f| vectors(f).len()).sum(); - assert_eq!(report.checks.len(), pack::FILES.len() + total); + assert_eq!(report.checks.len(), pack::FILES.len() + total + 2); } #[test] @@ -48,16 +50,13 @@ fn the_published_pack_is_exactly_what_the_reference_generates() { } } -/// Copy the published pack to `dir`, apply `mutate` to one vector, and return -/// the names of the vectors the verifier then rejects. -fn failures_after(dir: &Path, file: &str, name: &str, mutate: impl Fn(&mut Value)) -> Vec { +/// Write the published pack to `dir` with `edit` applied to the files' +/// parsed contents, and return the names of the checks the verifier fails. +fn failures_with(dir: &Path, edit: impl Fn(&str, &mut Vec)) -> Vec { std::fs::create_dir_all(dir).unwrap(); for f in pack::FILES { let mut items = vectors(f); - if f == file { - let v = items.iter_mut().find(|v| v["name"] == name).expect(name); - mutate(v); - } + edit(f, &mut items); std::fs::write(dir.join(f), serde_json::to_string_pretty(&items).unwrap()).unwrap(); } verify_pack(dir) @@ -67,6 +66,10 @@ fn failures_after(dir: &Path, file: &str, name: &str, mutate: impl Fn(&mut Value .collect() } +fn scratch(name: &str) -> std::path::PathBuf { + Path::new(env!("CARGO_TARGET_TMPDIR")).join(name) +} + /// Falsification: the verifier is not vacuous. Each planted defect — a wrong /// value, a wrong error class, a negative vector turned positive — must be /// caught, and caught on the vector it was planted in. @@ -97,6 +100,11 @@ fn the_verifier_catches_a_planted_defect_in_every_file() { "canon-004-floats-forbidden", Box::new(|v| v["input_cbor_hex"] = json!("a16a65786163745f68616c6601")), ), + ( + "canonicalisation_vectors.json", + "canon-020-simple-value-refused", + Box::new(|v| v["input_cbor_hex"] = json!("f6")), + ), ( "said_vectors.json", "said-008-spec-worked-example", @@ -141,13 +149,23 @@ fn the_verifier_catches_a_planted_defect_in_every_file() { Box::new(|v| v["error"] = json!("malformed_signature")), ), ( - "lockstep_vectors.json", - "lockstep-002-supermajority-commit", - Box::new(|v| v["expected_dissent"] = json!([])), + "signature_vectors.json", + "sig-007-non-canonical-key-rejected", + // The canonical spelling of the same point: now a well-formed key + // that merely fails to verify, so the published class is wrong. + Box::new(|v| { + v["verifying_key"] = + json!("0300000000000000000000000000000000000000000000000000000000000000") + }), ), ( "lockstep_vectors.json", - "lockstep-004-three-nodes-one-liar-split", + "lockstep-002-supermajority-commit", + Box::new(|v| v["expected_dissent_logs"]["node_1"] = json!([])), + ), + ( + "lockstep_vectors.json", + "lockstep-004-three-nodes-one-diverged-split", Box::new(|v| v["expected_commit"] = json!(true)), ), ( @@ -155,12 +173,17 @@ fn the_verifier_catches_a_planted_defect_in_every_file() { "lockstep-002-supermajority-commit", Box::new(|v| v["expected_outcomes"]["node_4"] = json!("commit_with_dissent")), ), + ( + "lockstep_vectors.json", + "lockstep-006-transport-liar-is-logged", + Box::new(|v| v["expected_outcomes"]["node_4"] = json!("minority_divergence")), + ), ( "gossip_vectors.json", "gossip-002-said-conflict-rejected", Box::new(|v| { v["expected_results"][1] = - json!({"outcome": "accepted", "said": v["expected_results"][0]["said"]}) + json!({"outcome": "duplicate", "said": v["expected_results"][0]["said"]}) }), ), ( @@ -168,14 +191,29 @@ fn the_verifier_catches_a_planted_defect_in_every_file() { "gossip-005-non-canonical-copy-rejected", Box::new(|v| { let s = v["deliveries"][0]["claimed_said"].clone(); + let b = v["deliveries"][0]["cbor_hex"].clone(); v["expected_results"][0] = json!({"outcome": "accepted", "said": s}); - v["expected_held"] = json!([s]); + v["expected_held"] = json!([{"said": s, "cbor_hex": b}]); + }), + ), + ( + "gossip_vectors.json", + "gossip-001-cross-node-propagation", + Box::new(move |v| { + let h = v["expected_held"][0]["cbor_hex"] + .as_str() + .unwrap() + .to_owned(); + v["expected_held"][0]["cbor_hex"] = json!(flip_last(&h)); }), ), ]; - let root = Path::new(env!("CARGO_TARGET_TMPDIR")).join("planted"); for (i, (file, name, mutate)) in plants.iter().enumerate() { - let failed = failures_after(&root.join(i.to_string()), file, name, mutate); + let failed = failures_with(&scratch(&format!("planted/{i}")), |f, items| { + if f == *file { + mutate(items.iter_mut().find(|v| v["name"] == *name).expect(name)); + } + }); assert_eq!( failed, vec![name.to_string()], @@ -184,11 +222,84 @@ fn the_verifier_catches_a_planted_defect_in_every_file() { } } +/// Falsification: a pack cannot pass by losing vectors. +#[test] +fn an_incomplete_pack_fails() { + // Every file emptied. + let failed = failures_with(&scratch("incomplete/empty"), |_, items| items.clear()); + for f in pack::FILES { + assert!(failed.contains(&"(file)".to_string()), "{f}"); + } + assert!(failed.contains(&"spec-cited vectors present".to_string())); + assert!(failed.contains(&"README error classes exercised".to_string())); + + // One vector the spec cites, deleted. + let failed = failures_with(&scratch("incomplete/said-010"), |f, items| { + if f == "said_vectors.json" { + items.retain(|v| !v["name"].as_str().unwrap().starts_with("said-010-")); + } + }); + assert_eq!(failed, vec!["spec-cited vectors present".to_string()]); + + // The only vector exercising a README error class, deleted. + let failed = failures_with(&scratch("incomplete/class"), |f, items| { + if f == "canonicalisation_vectors.json" { + items.retain(|v| v["name"] != "canon-020-simple-value-refused"); + } + }); + assert_eq!(failed, vec!["README error classes exercised".to_string()]); +} + +/// Falsification: a vector that names an error class the README does not +/// list is caught, so the README stays the full list. +#[test] +fn an_undocumented_error_class_fails() { + let failed = failures_with(&scratch("undocumented"), |f, items| { + if f == "canonicalisation_vectors.json" { + let mut extra = items + .iter() + .find(|v| v["name"] == "canon-018-trailing-bytes-refused") + .unwrap() + .clone(); + extra["name"] = json!("canon-099-made-up"); + extra["error"] = json!("made_up_class"); + items.push(extra); + } + }); + assert!(failed.contains(&"README error classes exercised".to_string())); + assert!(failed.contains(&"canon-099-made-up".to_string())); +} + +#[test] +fn a_pack_with_a_duplicated_json_member_fails() { + let dir = scratch("duplicate-member"); + std::fs::create_dir_all(&dir).unwrap(); + for f in pack::FILES { + let mut body = published(f); + if f == "lockstep_vectors.json" { + // Name a member twice; a plain JSON parser would keep the last. + body = body.replacen( + "\"node_count\": 4,", + "\"node_count\": 4,\n \"node_count\": 4,", + 1, + ); + } + std::fs::write(dir.join(f), body).unwrap(); + } + let report = verify_pack(&dir); + let failed: Vec<&str> = report.failures().iter().map(|c| c.file.as_str()).collect(); + assert_eq!(failed, vec!["lockstep_vectors.json"]); +} + #[test] fn the_said_pack_holds_the_suite_the_spec_names() { - // Identity spec §2.5: "the empty-map case, the one-field case, the worked - // example above, several envelope-shaped objects from treatise §8, and one - // adversarial case where a non-canonical CBOR encoding ...". + let cited = spec_cited_said_vectors().unwrap(); + for n in 1..=11 { + assert!( + cited.contains(&format!("said-{n:03}")), + "the spec no longer cites said-{n:03}" + ); + } let keys = |name: &str| -> BTreeSet { by_name("said_vectors.json", name)["input"] .as_object() @@ -201,31 +312,6 @@ fn the_said_pack_holds_the_suite_the_spec_names() { assert_eq!(keys("said-006-only-said-field"), set(&["d"])); assert_eq!(keys("said-007-one-field-per-key"), set(&["d", "k"])); assert_eq!(keys("said-008-spec-worked-example"), set(&["d", "n"])); - for n in 1..=5 { - let name = vectors("said_vectors.json")[n - 1]["name"] - .as_str() - .unwrap() - .to_owned(); - assert_eq!( - keys(&name), - set(&["d", "provenance", "ownership_chain", "cargo", "joule_cost"]), - "{name} has the five fields of treatise §8.1 plus its SAID" - ); - } - let adversarial: Vec = vectors("said_vectors.json") - .into_iter() - .filter(|v| { - matches!( - v["error"].as_str(), - Some("non_shortest_integer" | "unsorted_map_keys") - ) - }) - .map(|v| v["name"].as_str().unwrap().to_owned()) - .collect(); - assert!( - !adversarial.is_empty(), - "no adversarial non-canonical vector" - ); } #[test] @@ -240,6 +326,7 @@ fn every_canonical_rule_has_a_negative_vector() { "unsorted_map_keys", "duplicate_map_key", "float_forbidden", + "simple_value_forbidden", "tag_forbidden", "indefinite_length", "invalid_utf8", @@ -249,6 +336,10 @@ fn every_canonical_rule_has_a_negative_vector() { classes.contains(rule), "no canonicalisation vector is refused with {rule}" ); + assert!( + readme_error_classes().unwrap().contains(rule), + "README omits {rule}" + ); } } @@ -270,10 +361,6 @@ fn the_spec_prints_the_worked_example_the_pack_computes() { spec.contains("total 61 bytes"), "§2.4 must give the encoding's real length" ); - assert!( - !spec.contains("3f a1 7b 9c"), - "§2.4 still prints the old illustrative digest" - ); } #[test] diff --git a/smart-byte/spec/identity_and_key_rotation.md b/smart-byte/spec/identity_and_key_rotation.md index 67bdf7982..c8d614687 100644 --- a/smart-byte/spec/identity_and_key_rotation.md +++ b/smart-byte/spec/identity_and_key_rotation.md @@ -22,7 +22,7 @@ A **Self-Addressing IDentifier** is the cryptographic digest of a canonically-en 4. Encode the digest using the prefix-plus-bytewise-base32 encoding in §2.2. 5. Replace the placeholder in the object with the resulting SAID. -After step 5, the object is *self-certifying*: any party in possession of the object can run steps 1–4 against a copy with the SAID field reset to the placeholder, compute the digest, and confirm that the object's `d` field matches. Any modification of any other field — by even a single bit — produces a different digest, and the verification fails. +After step 5, the object is *self-certifying*: any party in possession of the object can run steps 1–4 against a copy with the SAID field reset to the placeholder, compute the digest, and confirm that the object's `d` field matches. Any modification of any other field — by even a single bit — produces a different digest, and the verification fails. A party that holds the object as bytes it received first confirms they are the object's canonical encoding (§2.3) and refuses them otherwise; it never re-encodes received bytes into canonical form and verifies the result (§2.5, §2.6). This is the same content-addressing pattern used for envelope identity in treatise §8.2; the SAID construction generalizes it to *any* object that needs a content-derived identifier including those that must reference themselves (key-event log entries, anchors, attestations). @@ -45,10 +45,11 @@ The canonical encoding for SAID computation is CBOR (RFC 8949) with deterministi - Strings use the shortest length-prefix. - Floats are forbidden. Any value that would be a float in another encoding must be represented as a fixed-point integer pair. - CBOR tags are forbidden. Tagged values change parser behavior and undermine determinism. +- Simple values other than `false`, `true` and `null` are forbidden. The data model has nothing else to decode them to. - Indefinite-length items are forbidden. -- The encoded form is the *only* form over which the digest is computed; serializations that round-trip through other formats (JSON, MessagePack) must re-encode to canonical CBOR before digest computation. +- The encoded form is the *only* form over which the digest is computed. An object held in another format (JSON, MessagePack) is encoded to canonical CBOR before its digest is computed. CBOR bytes received from another party must already be canonical: a verifier refuses any other encoding and never re-encodes it (§2.5, §2.6). -These are the deterministic encoding rules from RFC 8949 §4.2.1 with the tag-and-float prohibitions made explicit. They are also the rules used by envelope serialization in treatise §8 — the choice is intentional, so that a SAID computed over an envelope-shaped object yields the same identifier whether the SAID is being computed by Smart Byte's envelope-identity machinery or its key-event log machinery. +These are the deterministic encoding rules from RFC 8949 §4.2.1 with the tag, float and simple-value prohibitions made explicit. They are also the rules used by envelope serialization in treatise §8 — the choice is intentional, so that a SAID computed over an envelope-shaped object yields the same identifier whether the SAID is being computed by Smart Byte's envelope-identity machinery or its key-event log machinery. ### 2.4 Worked example @@ -108,11 +109,22 @@ The final object, with the placeholder replaced, is: These are the values of vector `said-008-spec-worked-example` in [`../conformance/v1/said_vectors.json`](../conformance/v1/said_vectors.json), generated by the reference implementation and reproduced by an independent second implementation (`../conformance/crosscheck/verify_v1.py`). -Any verifier in possession of the object replaces `d` with 53 `#` characters, re-encodes canonically, computes BLAKE3-256, base32-encodes with the `E` prefix, and confirms equality with the asserted `d`. +A verifier holding the object's bytes confirms they are canonical, replaces `d` with 53 `#` characters, encodes the result canonically, computes BLAKE3-256, base32-encodes it with the `E` prefix, and confirms equality with the asserted `d`. ### 2.5 Test-vector hook -The full set of canonical worked examples — placeholder bytes, canonical CBOR bytes, BLAKE3 digest, final SAID — lives in `../conformance/v1/said_vectors.json`. Implementers MUST pass every vector. The suite covers the empty-map case (`said-006`), the one-field case (`said-007`, a §3.4 per-key object), the worked example above (`said-008`), five envelope-shaped objects from treatise §8 (`said-001` to `said-005`), and two adversarial cases where a non-canonical CBOR encoding produces a different digest (`said-009`, `said-010`), to verify that implementations reject non-canonical input rather than silently re-encoding it or hashing it as received. `said-011` is a canonical object that states a SAID its content does not hash to. The canonical-encoding rules of §2.3 are exercised one by one in `../conformance/v1/canonicalisation_vectors.json`. +The full set of canonical worked examples — placeholder bytes, canonical CBOR bytes, BLAKE3 digest, final SAID — lives in `../conformance/v1/said_vectors.json`. Implementers MUST pass every vector. The suite covers the empty-map case (`said-006`), the one-field case (`said-007`, a §3.4 per-key object), the worked example above (`said-008`), five nested objects built from an envelope's members (`said-001` to `said-005`), and two adversarial cases where a non-canonical CBOR encoding produces a different digest (`said-009`, `said-010`), to verify that implementations reject non-canonical input rather than silently re-encoding it or hashing it as received. `said-011` is a canonical object that states a SAID its content does not hash to. The canonical-encoding rules of §2.3 are exercised one by one in `../conformance/v1/canonicalisation_vectors.json`. + +### 2.6 Receiving SAID-addressed objects + +An object that arrives from another party — gossiped (treatise §11.4) or as the answer to a request by SAID (treatise §21.4) — is accepted in four steps, in this order: + +1. Its bytes are the canonical encoding of the object (§2.3). If they are not, the object is refused; it is never re-encoded. +2. It hashes to the SAID it states in `d`. +3. That SAID is the one it was announced or requested under. +4. Only then: if the SAID is already held, the object is a duplicate and changes nothing. + +Verification precedes deduplication, so a forged object announced under a SAID the receiver already holds is reported as a SAID mismatch — the substitution attempt is visible — instead of being absorbed as a duplicate. No first-come rule is needed: two different canonical objects cannot share a SAID short of a BLAKE3 collision. `../conformance/v1/gossip_vectors.json` exercises each step. --- @@ -184,17 +196,17 @@ Notes on individual fields: - `s` starts at `0` for `icp` and increments by exactly 1 per event — no gaps, no reuse. - `p` is the empty string for `icp` and the previous event's `d` for every other event. - `kt` is an unweighted integer threshold; KERI's weighted-threshold extension is reserved for a later revision. -- Each entry in `k` is a public-key encoding prefixed with the one-byte algorithm identifier from treatise §8.3 (`0x01` for ed25519; remaining range reserved for post-quantum schemes such as ML-DSA). Order matters; the same key in a different position produces a different `n` commitment and breaks the chain. +- Each entry in `k` is a byte string: the one-byte algorithm identifier from treatise §8.3 (`0x01` for ed25519; the remaining range is reserved for post-quantum schemes such as ML-DSA) followed by the public key. Order matters; the same key in a different position produces a different `n` commitment and breaks the chain. - `nt` is committed in advance so an adversary cannot lower the threshold without revealing pre-rotated keys. - Each entry in `n` is the SAID of the corresponding next signing key per §3.4. An empty `n` signals the controller does not intend to rotate further; thereafter only `ixn` events are accepted. - `a` is populated for `ixn` and empty for `icp` and `rot`. ### 3.4 Per-key SAID derivation -Each entry in `n` is the SAID of a tiny CBOR map containing the corresponding next signing key, computed per §2 with the placeholder rules. Specifically, for each next-key `K_i` (the full prefixed public-key encoding the controller intends to use after the next rotation), the entry `n[i]` is: +Each entry in `n` is the SAID of a tiny CBOR map containing the corresponding next signing key, computed per §2 with the placeholder rules. Specifically, for each next-key `K_i` (the algorithm-prefixed key byte string of §3.3 the controller intends to use after the next rotation), the entry `n[i]` is: ``` -SAID({"d": "", "k": ""}) +SAID({"d": "", "k": h''}) ``` with `d` initialized to the 53-`#` placeholder, the map canonically encoded per §2.3, the digest computed per §2.2, and `d` replaced with the resulting SAID. This per-key SAID is what appears in the previous event's `n` array; at rotation, the verifier recomputes the per-key SAID for each `k[i]` in the new event and confirms it matches `n[i]` from the previous event, position-wise. @@ -213,7 +225,7 @@ This separation lets a verifier confirm the event's SAID independently of signat Given a sequence of events and a current controller AID, a verifier accepts the log if and only if every check below passes for every event in order from `s: 0` to the latest event seen. -1. **SAID verification.** Replace the event's `d` field with the 53-character placeholder, canonically re-encode per §2.3, compute BLAKE3-256, base32-encode with the `E` prefix, and confirm equality with the asserted `d`. Reject the event if the recomputed SAID does not match. +1. **SAID verification.** Confirm the event's bytes are canonical (§2.3) and reject them otherwise (§2.6). Replace the event's `d` field with the 53-character placeholder, encode canonically, compute BLAKE3-256, base32-encode with the `E` prefix, and confirm equality with the asserted `d`. Reject the event if the recomputed SAID does not match. 2. **Sequence verification.** Confirm the event's `s` is exactly the previous event's `s` + 1. For the inception event, confirm `s` is `0`. Reject the event on any gap, repeat, or out-of-order value. @@ -253,6 +265,7 @@ The Rust reference implementation of this section lives in the `smart-byte-rs` w - `smart-byte-rs/crates/smart-byte-core/src/canonical.rs` — the canonical CBOR encoding of §2.3, and a strict decoder that refuses non-canonical input instead of re-encoding it. - `smart-byte-rs/crates/smart-byte-core/src/said.rs` — SAID derivation and verification (§2.1, §2.2), placeholder substitution, and per-key SAID derivation (§3.4). +- `smart-byte-rs/crates/smart-byte-core/src/inbox.rs` — receiving SAID-addressed objects (§2.6). - `smart-byte-rs/crates/smart-byte-conformance` — the conformance-vector harness referenced in §2.5. It generates `../conformance/v1` from the two modules above and checks the published pack under `cargo test`. - `smart-byte-rs/crates/smart-byte-keri-witness` — the key-event log: event type definitions, the wrapper that pairs events with signatures, the verifier algorithm in §4, and helpers for emitting `icp`, `rot`, and `ixn` events. **Its event SAIDs do not follow §2 yet**: it hashes a general-purpose CBOR serialization with a zeroed placeholder, not the canonical encoding with the 53-`#` placeholder, so its identifiers are not interoperable with this section. -- 2.54.0 (Apple Git-157)