wai: bearer-receipts §13, reconciliations and holdings records over a return path of a few bytes #637

Merged
dcharlot merged 2 commits from wai/compact-return-profile into main 2026-11-06 13:44:54 -05:00
16 changed files with 5738 additions and 2 deletions

View file

@ -0,0 +1,28 @@
# step: compact return profile · wai-rs and compact_return_verify.py
# order: 535
# features: bearer_receipts,holdings
# paths: wai/extensions/bearer-receipts.md
#
# The compact return profile (extensions/bearer-receipts.md §13): the reference module's unit
# tests (a uint's one spelling, an instant's, the whole bitmap, a message reassembled in any
# order with each defect refused, the bounds, and the corpus reaching every stated outcome and
# failing a changed one) and tests/compact_return.rs (vectors.json regenerates byte for byte
# from its fixed seeds and no other file is there, wai-rs gives every case its stated outcome
# with every round trip holding both ways, §13.9 lists exactly the reference's codes, and
# §13.8's floors are the lengths of the aggregate corpus's reports, read with the reference's
# DAP reader); then the corpus's second implementation, compact_return_verify.py, written from
# the extension's text and sharing no code with wai-rs, which reads groups and objects with
# jwp_verify.py, claims with claim_verify.py, §5.3 with bearer_verify.py, holdings records with
# holdings_verify.py and DAP reports with telemetry_aggregate_verify.py (the claim verifier
# needs the `cryptography` package installed above), writes the references, the compact
# records, their expansion and writing, fragments and reassembly itself, checks the floors
# against telemetry-aggregation's table and corpus, requires every mutated copy of the corpus
# to fail, and drops each of its own checks in turn; run isolated (-I).
#
# The paths this step brings to the workflow's trigger, and why:
# wai/extensions/bearer-receipts.md: tests/compact_return.rs requires §13.9 to list the compact
# profile's codes and §13.8 its floors, and compact_return_verify.py reads both.
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib -- compact_return::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test compact_return)
python3 -I live-conformance/compact_return_verify.py live-conformance/compact-return

View file

@ -0,0 +1,40 @@
### Reconciliations and holdings records over a return path of a few bytes
- **The compact return profile** ([extensions/bearer-receipts.md](extensions/bearer-receipts.md)
§13): a reconciliation (§5) and a holdings record ([holdings](extensions/holdings.md) §4) in a
binary form for a return path whose messages hold few bytes. It names by reference what the
receiver already holds: the group (8 bytes of SHA-256 over its link hash), the bearer instance
among that group's (4 bytes over its `bearer_id`) and the signer (8 bytes over its key); it writes
integers in LEB128 and `issued_at` as seconds. A reference that resolves to no value, or to more
than one, is refused, never resolved by choosing. A whole reconciliation carries no bitmap and its
root is the group's; a partial one carries its bitmap, and its root is derived from the object
receipts the receiver holds.
- **Exact both ways.** A compact record expands, against the receiver's context, to the record's
own bytes, a claim's canonical form or a holdings record's `WHD1`, which are then verified by
their own text under the signature the compact record carried: the compact form states nothing of
its own. A record has one compact form against a context. One that has none (an `issued_at` with a
fraction or an offset, a group or instance the receiver does not hold, a `leaf_count` not the
group's, a root its objects do not give, a hybrid claim) is refused `compact_not_encodable` and
travels in its own form; nothing is sized by a claim's own `leaf_count` before it is found to be
the group's.
- **Fragments.** A message is the compact record and a 4-byte check, the first bytes of SHA-256 over
the record, cut into frames under a 3-byte header: the sender's number for the message, the index
and the count. Reassembly places fragments by index in any order of arrival and holds a fragment
delivered twice, byte for byte, once. It refuses a fragment malformed, foreign (another number or
count), repeated (other data at its index) or missing, and, by the check, one reordered or mixed
in under the same header.
- **DAP reports are never fragmented.** A task's floor is the length of its longest report; a path
whose frame is below it carries none of the task's reports, and a sink holds them, or gives them to
a node with a path that carries them, which uploads them unchanged. The floors of
telemetry-aggregation's measures 1 to 4 are stated (552, 824, 2,664 and 648 bytes) and checked
against the reports of its corpus.
- **Beside it**: jwp-receipts §2.7 and holdings §1 point to the profile, and the CI's check of each
feature list alone checks `bearer_receipts,holdings`.
- **The corpus** [`live-conformance/compact-return/`](live-conformance/compact-return/): round trips
of reconciliations in each signer form and over each bearer class and of holdings records,
expansions (every form a reader takes, a changed byte refused by the record's own signature, a
control for each reader rule of both records, references unknown and ambiguous, an instance held
for other groups only, a holder's key under two `signer_id`s), records with no compact form,
messages cut into frames of several sizes, and reassemblies with each defect. The reference is the
`compact_return` module (features `bearer_receipts` and `holdings`); `compact_return_verify.py` is
a second implementation.

View file

@ -17,6 +17,10 @@
> `bearer_receipts`), with the two classes' bodies in `claim_receipt::bearer`; the corpus is
> [`live-conformance/bearers/`](../live-conformance/bearers/), and its second implementation
> is `live-conformance/bearer_verify.py`.
> §13 writes a reconciliation and a holdings record compactly for return paths whose messages
> hold few bytes; its reference is the `compact_return` module (features `bearer_receipts` and
> `holdings`), its corpus [`live-conformance/compact-return/`](../live-conformance/compact-return/)
> and its second implementation `live-conformance/compact_return_verify.py`.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.
>
> Texts read for this revision, each at the edition named: ETSI TS 103 769 V1.2.1 (2024-11);
@ -32,7 +36,9 @@ In scope: the bearer classes (§2); why the bearer is declared beside the receip
it (§3); the transmission claim (§4) and the reconciliation claim (§5), their bodies, rules
and context checks; the verdict a verifier derives for one group from its receipts and those
claims (§6); the upper bound on a receiver's share of a shared transmission's energy (§7);
verification with no return path (§8); and the mappings onto bearers (§9).
verification with no return path (§8); the mappings onto bearers (§9); and a compact form of
reconciliations and holdings records for return paths whose messages hold few bytes, carried in
fragments (§13).
When each object passes on a bearer with no return path, and with which forward error
correction, is [carousel](carousel.md)'s: a publisher's `schedule` claim beside the transmission
@ -570,3 +576,318 @@ mutates copies of the corpus and requires every copy to fail.
controls the bearer can withhold revisions — a revocation among them — but cannot add a key
to an anchored origin. A sink that has had no path for long holds a stale log, and its
confirmation reports the head it used.
## 13. The compact return profile
A path back from a sink or a node may carry only a few bytes in each message: a radio frame, a
text message, a satellite uplink. A reconciliation (§5) is a JSON claim that writes its group's
link hash, its root, its signer's key and its signature in hexadecimal, beside the instance's URI;
a holdings record ([holdings](holdings.md) §4) names its holder and each group by 32 bytes. This
profile writes both records in a binary form that names by short references what the receiver
already holds — the group, the bearer instance, the signer — and carries a message too long for
one frame in fragments, with a check over the whole message.
The compact form is an encoding, never a statement of its own. A receiver expands a compact
record, against what it holds, into the record's own bytes and verifies that record by its own
text; the signature a compact record carries is the record's. Nothing here changes what a
reconciliation or a holdings record states, how either is verified, or what a verifier derives
from it (§6, holdings §5).
### 13.1 Terms
- **Path**: what carries messages from a **sender** (a sink, or a node) to a **receiver** (the
distributor that reconciles, or a node that gathers for it). Its **frame** is the most bytes
one of its messages carries.
- **Context**: what the receiver holds that a compact record names. Nothing a compact record
carries adds to it.
- **Groups**: group receipts that pass jwp-receipts §3 steps 1 and 2, each with the object
receipts held under it that pass §3 step 5 against it, and the `bearer_id` of each instance
of the group the receiver holds a declaration of (§4).
- **Signers**: keys, each with a `signer_id`: keys the receiver confirmed under jwp-receipts
§7, or keys its senders enrolled with it.
- **Compact record**: a reconciliation (§13.3) or a holdings record (§13.4) in this profile's
form, at most 65,531 bytes. **Message**: a compact record followed by its **check**, the first 4
bytes of SHA-256 over the compact record. **Fragment**: one frame's part of a message (§13.6).
A fixed-size integer is unsigned and big-endian.
### 13.2 Uints, instants and references
- A **uint** is an unsigned integer in LEB128: seven bits a byte, least significant first, the
high bit set in every byte but the last. It is written in the fewest bytes, so a last byte of
0x00 follows no other byte, and its value is at most 2^53 − 1, so it has at most 8 bytes.
- An **instant** is a uint of seconds since 1970-01-01T00:00:00Z, leap seconds not counted, at
most 253,402,300,799 (9999-12-31T23:59:59Z). It stands for the RFC 3339 date-time of that
second written `YYYY-MM-DDTHH:MM:SSZ`, with `T` and `Z` in capitals, no fraction and a second
below 60. A date-time written in any other way has no instant.
- The **reference** of a value is the first *n* bytes of SHA-256 over it: a group's is 8 bytes
over its link hash (32 bytes); a bearer instance's, 4 bytes over its `bearer_id` in UTF-8; a
signer's, 8 bytes over its key (32 bytes).
- A reference **resolves** to the one value of its kind in the context that has it: a group
among the groups; a bearer instance among the instances of the group the record names; a
signer among the signers — a key with its `signer_id` for a reconciliation, a key for a
holdings record, which names no `signer_id`. Values that are equal are one value. A
reference that no value has is refused `compact_reference_unknown`, and one that more than
one value has, `compact_reference_ambiguous`: neither is resolved by choosing.
### 13.3 A compact reconciliation
| size | field | meaning |
|---|---|---|
| 1 | `kind` | 0x01 |
| 1 | `form` | below |
| 8 | `group` | the reference of the group: the claim's `subject` |
| 4 | `bearer` | the reference of the bearer instance: the body's `bearer_id` |
| uint | `seq` | the claim's `seq` |
| uint | `issued_at` | the claim's `issued_at`, as an instant (§13.2) |
| 32, 32 + uint + *n*, or 8 | `signer` | by the signer form, below |
| 32 or 0 | `parent` | the claim's `parent_receipt_hash`, where `form` says the claim has one |
| ⌈`leaf_count` / 8⌉ or 0 | `received` | the bitmap of §5.1, where `form` says the reconciliation is partial; `leaf_count` is the group's |
| 64 | `sig` | the claim's `sig` |
`form` is read by its bits:
| bits | meaning |
|---|---|
| `form & 0x03` | the body's `bearer`: 0 `bidirectional`, 1 `multicast`, 2 `unidirectional` |
| `form & 0x04` | set: **whole**, every object of the group received, and no `received` follows; clear: **partial** |
| `(form >> 3) & 0x03` | the signer form: 0, the key (32 bytes), whose Ed25519 `did:key` (jwp-receipts §7.1) is the `signer_id`; 1, the key, then a uint *n* and the `signer_id` in *n* bytes of UTF-8; 2, the reference of the signer (8 bytes) |
| `form & 0x20` | set: a `parent` follows; clear: `parent_receipt_hash` is `null` |
| `form & 0xc0` | 0 |
A reader reads exactly the bytes it is given, field by field in the order of the table, and
refuses them `compact_malformed` when they are more than 65,531 bytes, end before a field or go
on after `sig`, or when:
1. `form` sets a bit of 0xc0, or names the bearer 3 or the signer form 3;
2. a uint breaks §13.2, or `issued_at` is past its bound;
3. in the signer form 1, the `signer_id` is not UTF-8, or is the key's own `did:key`, which is
the signer form 0;
4. a partial `received` sets no bit, sets a bit at or past `leaf_count`, or sets every bit below
it, which is the whole form.
It resolves `group` as it reads it, since `received`'s length is the group's; then `bearer` among
that group's instances; and, in the signer form 2, `signer` (§13.2).
Its **expansion** is the reconciliation claim in its canonical form (jwp-receipts §2.4.3), with:
- `kind` `claim`, `class` `reconciliation`, `subject_type` `receipt`, and `subject` the link hash
of the group resolved;
- `seq`, `issued_at` (the instant's date-time), `signer_pubkey`, `signer_id`,
`parent_receipt_hash` and `sig` from the fields;
- a body of `bearer` from `form`, `bearer_id` the instance resolved, `leaf_count` the group's,
`attests` `received`, and:
- whole: `received` the bitmap that sets every bit below `leaf_count`, and `received_root` the
group's `root_hash`;
- partial: `received` as carried, and `received_root` the root jwp-receipts §2.2 computes over
the leaves of the object receipts the context holds at its set bits, in increasing order, each
leaf in its object receipt's form. A set bit at which the context holds no object receipt is
refused `compact_objects_not_held`.
The expansion is then verified as every reconciliation is: jwp-receipts §3.1, and §5 here. A
compact record whose bytes were changed, where it still reads, expands to a claim its signer did
not sign.
### 13.4 A compact holdings record
| size | field | meaning |
|---|---|---|
| 1 | `kind` | 0x02 |
| 1 | `form` | 0: the holder's key follows (32 bytes); 1: the reference of the holder follows (8 bytes) |
| 32 or 8 | `holder` | the holder |
| uint | `as_of_ms` | the record's `as_of_ms` |
| uint | `window_ms` | its `expires_ms` less its `as_of_ms` |
| uint | `n` | its entries' count, at most 2^32 − 1 |
| … | `entries` | `n` entries, below, in the record's order |
| 64 | `sig` | the record's `sig` |
An entry:
| size | field | meaning |
|---|---|---|
| 1 | `form` | `form & 0x01` set: every object of the group (holdings §4.2's form 1), and no `held` follows; `form & 0x02` set: the group by reference; every other bit 0 |
| 8, or 32 + uint | `group` | by reference: the reference of the group, whose `leaf_count` is the group's; else the group's link hash, then its `leaf_count` as a uint |
| ⌈`leaf_count` / 8⌉ or 0 | `held` | the bitmap of holdings §4.2, in its form 0 |
A reader reads exactly the bytes it is given, in the order of the tables, resolving each
reference as it reads it, and refuses them `compact_malformed` when they are more than 65,531
bytes, end before a field or go on after `sig`, or when a `form` is not one the tables give, a
uint breaks §13.2, or `n` is past its bound. Its **expansion** is the holdings record of holdings
§4.1: `WHD1`, version 1, the holder's key, `as_of_ms`, `as_of_ms` + `window_ms`, `n`, each
entry's link hash, `leaf_count`, form and `held`, and `sig`. The expansion is then read as every
holdings record is, by holdings §4.4, and holdings §5 applies to it.
### 13.5 Writing a record compactly
The compact form of a record **against a context** — what its sender knows its receiver holds —
is the one the rules below give, and its expansion against that context is the record, byte for
byte. A reader takes every form: a value written inline expands as its reference would.
- **A reconciliation** has a compact form when it reads under jwp-receipts §3.1 steps 1 and 2 as
a claim of class `reconciliation` without the hybrid members of
[pq-signatures](pq-signatures.md) §3.1, its text is its canonical form, its `issued_at` has an
instant, and its expansion against the context is that record: the context holds its group
and, for that group, its instance; its `leaf_count` is the group's; and its `received` and
`received_root` are ones §13.3 reads and derives. Its signer is written by
reference where the context holds its key with its `signer_id` and the reference resolves to
them, else in the form 0 where its `signer_id` is its key's `did:key`, else in the form 1.
- **A holdings record** has a compact form when it reads under holdings §4.4, its signature
verifying. Its holder is written by reference where the context holds its key and the reference
resolves to it, and each entry's group by reference where the context holds that group with the
entry's `leaf_count` and the reference resolves to it; each is written inline otherwise.
A record that has no compact form is refused `compact_not_encodable`, and travels in its own
form. A sink that sends its reconciliations over a path whose frame is small **SHOULD** sign them
with an `issued_at` that has an instant: to the second, in UTC.
### 13.6 Fragments
| size | field | meaning |
|---|---|---|
| 1 | `message` | the sender's number for the message, the same in each of its fragments |
| 1 | `index` | the fragment's position in the message, from 0 |
| 1 | `count` | the message's fragments, 1 to 255 |
| … | `data` | at least one byte of the message |
A message is at most 65,535 bytes. A sender cuts it, in order, into fragments of the path's frame,
the last holding what remains: the fragments' `data`, in increasing `index`, are the message. It
numbers its messages on one path modulo 256, and does not use a number again while a fragment
under it may still arrive. A path whose frame is 3 bytes or fewer carries no fragment, and a
message that would need more than 255 fragments of a path's frame is not sent over that path.
### 13.7 Reassembly
The steps below are defined over a set of fragments taken as one message's. A receiver forms that
set from the fragments it receives, by their sender, as the path addresses it, and by `message`;
how it forms the set is not a step, and refuses nothing. A set formed by `message` holds no
fragment under another number, and step 2 refuses one in a set formed otherwise. Fragments that
are the same bytes are one fragment of the set, so a fragment the path delivers twice is held
once. Given a message's fragments, in any order, a receiver checks, in this order:
1. each fragment has at least 4 bytes, a `count` of at least 1 and an `index` below its `count`
(`fragment_malformed`);
2. each has the first's `message` and `count` (`fragment_foreign`);
3. no two that differ have one `index` (`fragment_repeated`);
4. there are `count` of them (`fragment_missing`); a message of which no fragment is held is
missing;
5. their `data`, in increasing `index`, the message, is at most 65,535 bytes
(`fragment_malformed`);
6. the message has more than 4 bytes, and its last 4 are the check over the bytes before them
(`fragment_check_failed`).
The bytes before the check are the compact record, read by its `kind`: §13.3 for 0x01, §13.4 for
0x02, and `compact_malformed` for any other.
Fragments arrive in any order and are placed by their `index`. A fragment placed at a position
other than the one it was cut from — its `index` changed, or the data of two fragments exchanged —
is a **reordered** fragment. Where the headers still show a whole message, the check refuses the
message it is in (step 6), as it refuses a fragment of another message that carries this
message's `message` and `count`: steps 2 to 4 find what the headers show, and step 6 what they do
not.
### 13.8 Aggregated measurement on a small frame
A report of [telemetry-aggregation](telemetry-aggregation.md), a DAP report, is not a record of
this profile, and this profile never cuts one into fragments. A sink uploads each report whole to
the leader (telemetry-aggregation §5), and a report has no shorter form: its input shares are
encrypted to the aggregators. A task's **floor** is the length of its longest report. A sink
sends a task's reports only over a path that carries one whole, so a path whose frame is below a
task's floor carries none of its reports. Every report of a task of measures 1 to 4 of
telemetry-aggregation §3 has one length, its floor:
| measure | floor, bytes |
|---|---|
| 1 `outcomes` | 552 |
| 2 `stalls` | 824 |
| 3 `durations` | 2,664 |
| 4 `start_time` | 648 |
A sink whose every path is below a task's floor holds its reports until it has a path that
carries them, or gives them, over a short link, to a **node** that has one — a receiver's cache,
or a box that holds a carousel for the receivers near it. The node uploads each report as it
received it. It learns of each report its id, its time, its length and its task, as each
aggregator does (telemetry-aggregation §10), and which neighbour gave it, and nothing of its
measurement. A node **MAY** carry its neighbours' messages of this profile, or the records they
expand to, over its own path in the same way: a record a node carries is its sender's, unchanged,
and the node signs nothing of it.
### 13.9 Reasons
The codes this profile adds, each with where it is raised:
| reason | raised at |
|---|---|
| `compact_malformed` | §13.3, §13.4, §13.7 |
| `compact_reference_unknown` | §13.2 |
| `compact_reference_ambiguous` | §13.2 |
| `compact_objects_not_held` | §13.3 |
| `compact_not_encodable` | §13.5 |
| `fragment_malformed` | §13.7 steps 1 and 5 |
| `fragment_foreign` | §13.7 step 2 |
| `fragment_repeated` | §13.7 step 3 |
| `fragment_missing` | §13.7 step 4 |
| `fragment_check_failed` | §13.7 step 6 |
An expansion also raises the codes of its record's own verification: those of jwp-receipts §3.1
and §10 here for a reconciliation, and holdings §7's for a holdings record.
### 13.10 Conformance
[`live-conformance/compact-return/`](../live-conformance/compact-return/) holds the corpus,
written from fixed seeds by `wai_compact_return_vectors gen` as `vectors.json`. It has:
- **Round trips**: reconciliations whole and partial, over each bearer class, in each signer form,
with and without a parent, a group of labelled leaves and a bitmap whose last byte is part used
among them; and holdings records with their holder and groups by reference and inline. Each
case carries the record, the context and the compact record, written from the record and
expanded back to it, byte for byte, and the record's verdict under its own text.
- **Expansions**: every form a reader takes, a value the receiver holds written inline among
them; a compact record with a byte changed, which expands and is refused by its record's own
signature; one negative control for each rule of §13.3 and §13.4 but the bound of 65,531
bytes, which both implementations check themselves; references unknown and ambiguous, two
bearer instances that share one reference among them; a bearer instance the receiver holds for
other groups and not for the record's, which is unknown; a holder's key held under two
`signer_id`s, which resolves, since a holdings record names no `signer_id`; and a partial
reconciliation against a receiver that holds no object receipt.
- **Encodings**: records with no compact form, each refused, a reconciliation whose `leaf_count` is
2^53 − 1 beside a bitmap of one byte among them.
- **Messages and reassemblies**: compact records cut into frames of several sizes, one fragment
among them, and reassembled in order and in other orders, and with a fragment delivered twice,
byte for byte, which is held once; and a missing fragment, a repeated one (other data at its
`index`, and a fragment relabelled to another's `index`), a reordered one (data exchanged,
indices exchanged), a foreign one (another number, another count, and the same header from
another message), a changed byte, and malformed fragments.
A conforming verifier gives every case the outcome, the reason and everything else the corpus
states. `wai_compact_return_vectors verify` and the test `tests/compact_return.rs` check that the
file regenerates byte for byte, that `wai-rs` gives every case its outcome, that every reason of
§13.9 appears, that §13.9 lists exactly the reference's codes, and that §13.8's floors are the
lengths of the reports in telemetry-aggregation's corpus, read with the reference's DAP reader.
`live-conformance/compact_return_verify.py` is the second implementation, written from this text:
it shares no code with `wai-rs`; it reads groups, objects and leaves with the JWP corpus's second
implementation (`jwp-conformance/jwp_verify.py`), claims with the claim corpus's
(`claim-conformance/claim_verify.py`), §5.3 with `bearer_verify.py`, holdings records with
`holdings_verify.py` and DAP reports with `telemetry_aggregate_verify.py`, and writes §13.2 to
§13.8 itself; it checks the floors against telemetry-aggregation's own table and corpus; and it
mutates copies of the corpus and requires every copy to fail, and drops each of its own checks in
turn and requires the corpus to catch the weakened verifier.
### 13.11 Security and privacy considerations
- **The compact form carries no trust of its own.** It carries its record's signature, and the
receiver verifies the expansion by the record's own text. A reference that resolved to another
value would expand to another record, which that signature does not verify.
- **The check is not a signature.** Its four bytes of SHA-256 find what a fragment's header does
not show — a fragment misplaced, or mixed in from another message under the same header, and a
byte changed on the path — and authenticate nothing. Anyone on a path can make a message whose
check holds: what it expands to is refused unless its signer signed it.
- **What a reference discloses.** A reference names what the receiver holds, and says no less
than the record it stands for: anyone who holds the group receipts resolves a group's. A signer
by reference links a sender's messages as its key does. A sink that signs each reconciliation
with a fresh `did:key` (§12) sends its key inline, in the signer form 0.
- **Bounded work.** A message is at most 65,535 bytes in at most 255 fragments, a compact record is
read once, in order, and resolving a reference is a lookup. A receiver **SHOULD** bound how long
it holds the fragments of a message that has not completed; this profile sets no bound. A writer
sizes nothing by a reconciliation's own `leaf_count` before it has found it to be the group's
(§13.5): a record a node carries for a neighbour (§13.8) can state a `leaf_count` of 2^53 − 1
beside a bitmap of one byte and still read under jwp-receipts §3.1 steps 1 and 2.

View file

@ -27,6 +27,8 @@ Out of scope: how records travel (a neighbourhood's own announcements, a file be
carousel, a request); how a node chooses what to keep; how a neighbour is reached or what a link
costs, which the asker orders its neighbours by itself; and objects not delivered under a group
receipt, which have no holdings here.
A record's compact form, for a return path whose messages hold few bytes, is
[bearer-receipts](bearer-receipts.md) §13's; it expands to the record's own bytes.
Relationship to the core spec: `wai.delivery.holdings` is a **record** ([SPEC](../SPEC.md) §5).
It names no payload format and is not dispatchable.

View file

@ -1751,6 +1751,9 @@ value the JSON form gives it, and a verifier then reads and checks those
members by §3 unchanged. It is canonical: a receipt has one binary form, with
no optional encoding, so a writer reproduces the bytes it read. A receipt whose
JSON members a reader of §3 step 1 or step 4 refuses has no binary form.
A reconciliation (§2.4.15) has a compact form for return paths whose messages
hold few bytes, [bearer-receipts](bearer-receipts.md) §13, which a receiver
expands to the claim's canonical form before it verifies it.
#### 2.7.1. A group receipt

View file

@ -31,6 +31,7 @@ and the independent verifier of each, beside it.
| [`influence/`](influence/) | [influence](../extensions/influence.md) (`wai.media.influence`) | `wai_influence_vectors gen` | `influence_verify.py` |
| [`direction/`](direction/) | [direction](../extensions/direction.md) (`wai.media.direction`) | `wai_direction_vectors gen` | `direction_verify.py` |
| [`bearers/`](bearers/) | [bearer-receipts](../extensions/bearer-receipts.md) (`wai.bearer.receipts`) | `wai_bearer_vectors gen` | `bearer_verify.py` |
| [`compact-return/`](compact-return/) | [bearer-receipts](../extensions/bearer-receipts.md) §13 (the compact return profile) | `wai_compact_return_vectors gen` | `compact_return_verify.py` |
| [`carousel/`](carousel/) | [carousel](../extensions/carousel.md) (`wai.bearer.carousel`) | `wai_carousel_vectors gen` | `carousel_verify.py` |
| [`holdings/`](holdings/) | [holdings](../extensions/holdings.md) (`wai.delivery.holdings`) | `wai_holdings_vectors gen` | `holdings_verify.py` |
| [`access/`](access/) | [access-services](../extensions/access-services.md) (`wai.access.services`) | `wai_access_services_vectors gen` | `access_services_verify.py` |

View file

@ -0,0 +1,49 @@
# live-conformance/compact-return — reconciliations and holdings over a small frame
The corpus of [`extensions/bearer-receipts.md`](../../extensions/bearer-receipts.md) §13.10, in one
file, `vectors.json`.
**Provenance.** Generated here by `wai_compact_return_vectors gen` (`wai-rs`, features
`bearer_receipts` and `holdings`), from Ed25519 keys on fixed 32-byte seeds, constant times, and
envelopes whose payloads are opaque test bytes written by the generator; no media and no input from
outside this repository.
`keys` names every key by its role, as 64 lowercase hexadecimal digits: the publisher's
(`did:web:publisher.example#d1`), four sinks' and two holders'. `groups` holds each group receipt the
cases name, with its object receipts in order: `g4`, `g3` (labelled leaves), `g9`, `g20`, and `far`,
which no receiver holds. `contexts` holds each receiver's context (§13.1): for each group it holds,
the group's name, the positions of the object receipts it holds under it and the `bearer_id`s of
its instances; and its signers, each a `key` and a `signer_id`. `twin_bearers` holds two instances
of `g4` whose references are equal; `twin_signer`, sink B's key and the holder's each under two
`signer_id`s; `bearer_elsewhere`, the broadcast instance of `g9` and `g20` but not of `g4`.
- `records` (§13.3 to §13.5) are each a `kind` (`reconciliation` or `holdings`), the `record` in its
own form (a claim's canonical text, or a holdings record in hexadecimal), the `context` it is
written against and its `compact` record in hexadecimal. The record written compactly is
`compact`, `compact` expands to the record, and `outcome.verdict` is the record's verdict under
its own text: `accept`, or `read digest=…` for a holdings record.
- `expansions` (§13.2 to §13.4) are each a `compact` record and a `context`; `outcome` is `expand`
with the record's `kind`, the `record` and its `verdict`, or `refuse` and its `reason`.
- `encodings` (§13.5) are each a record with no compact form against its `context`; `outcome` is
`refuse` and `compact_not_encodable`.
- `messages` (§13.6) are each a compact `record`, the sender's `message` number and a `frame`;
`outcome.fragments` is the message cut into that frame, each fragment in hexadecimal. The
fragments reassemble to the record.
- `reassemblies` (§13.7) are each a set of `fragments`; `outcome` is `reassembled` and the compact
`record`, or `refuse` and its `reason`.
A reconciliation's verdict is jwp-receipts §3.1 steps 1 to 4, then step 6 by bearer-receipts §5.3
against the context's group and objects: `accept`, or `reject step=… reason=…`. Step 5, a signer's
chain, is the signer's claims' and not this profile's. A holdings record's is holdings §4.4: `read
digest=…`, or `reject reason=…`.
Everything is synthetic: keys attest nothing, and every origin is a reserved example domain. Each
case states the outcome it was written to have, declared in the generator, never copied from a
verifier. The bounds of 65,531 bytes for a compact record and 65,535 for a message are carried by no
vector (65 KB of hexadecimal); both implementations check them themselves.
`wai_compact_return_vectors verify` and `tests/compact_return.rs` check that the file regenerates
byte for byte and that `wai-rs` reaches every stated outcome; the test also checks §13.8's floors
against [`../aggregate/`](../aggregate/)'s reports. `../compact_return_verify.py` is the second
implementation, written from the extension's text, which also checks the floors, mutates copies of
the corpus and drops each of its own checks in turn and requires each to be caught.

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,763 @@
#!/usr/bin/env python3
"""Independent verifier of the compact return corpus (extensions/bearer-receipts.md §13).
Written from the extension's text. It shares no code with wai-rs, which generates the corpus, and
never runs it. It takes from the second implementations of the texts the profile builds on what
they define: group and object receipts, link hashes and leaves from the JWP corpus's
(`../jwp-conformance/jwp_verify.py`, jwp-receipts §2.2, §3); a claim's canonical form, the
Ed25519 `did:key` and a claim's steps 1 to 4 from the claim corpus's
(`../claim-conformance/claim_verify.py`, jwp-receipts §2.4.3, §3.1, §7.1); a reconciliation's
check against its group and objects from the bearer corpus's (`bearer_verify.py`,
bearer-receipts §5.3); the holdings record from the holdings corpus's (`holdings_verify.py`,
holdings §4.4); and the reading of a DAP report from the aggregation corpus's
(`telemetry_aggregate_verify.py`). Everything §13 adds is written here:
- §13.2: uints, instants and references, and how a reference resolves;
- §13.3, §13.4: the compact reconciliation and holdings record, read and expanded against a
receiver's context;
- §13.5: writing a record compactly, which it does itself and compares with the corpus's bytes;
- §13.6, §13.7: cutting a message into fragments and reassembling it;
- §13.8: each floor, checked against the reports of telemetry-aggregation's corpus, and each
measure's name against that extension's own table.
For every case it requires the outcome the corpus states, field for field; every round trip both
ways; every cut to reassemble to its record; that every reason of §13.9's table appears in the
corpus; then it mutates copies of the corpus, one defect each, and requires every copy to fail;
and it drops each of its own checks in turn and requires the corpus to catch the weakened
verifier.
python3 compact_return_verify.py <live-conformance/compact-return>
Exit status 0 only if every case verifies, every mutated copy fails and every weakened verifier
is caught. Python 3.8 or later; the `cryptography` package, which the claim verifier imports.
"""
import calendar
import copy
import datetime
import hashlib
import json
import os
import re
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
for d in ("claim-conformance", "jwp-conformance"):
sys.path.insert(0, os.path.join(HERE, "..", d))
sys.path.insert(0, HERE)
import bearer_verify as BV # noqa: E402
import claim_verify as CV # noqa: E402
import holdings_verify as HV # noqa: E402
import jwp_verify as JV # noqa: E402
import telemetry_aggregate_verify as TV # noqa: E402
EXTENSION = os.path.join(HERE, "..", "extensions", "bearer-receipts.md")
AGGREGATION = os.path.join(HERE, "..", "extensions", "telemetry-aggregation.md")
AGGREGATE_CORPUS = os.path.join(HERE, "aggregate", "vectors.json")
KIND_RECONCILIATION, KIND_HOLDINGS = 0x01, 0x02
HEADER, CHECK = 3, 4
MAX_MESSAGE = 65535
MAX_RECORD = MAX_MESSAGE - CHECK
MAX_SECONDS = 253402300799
MAX_INT = 2 ** 53 - 1
BEARERS = ("bidirectional", "multicast", "unidirectional")
# A weakened run drops one of these checks (see `WEAKENINGS`); None checks everything.
DROPPED = None
def drops(name):
return DROPPED == name
class Refused(Exception):
def __init__(self, reason):
super().__init__(reason)
self.reason = reason
def section(heading):
"""One `###` section of the extension, from its heading to the next heading."""
text = open(EXTENSION, encoding="utf-8").read()
at = text.index("\n### " + heading) + 1
end = text.find("\n#", at)
return text[at:end if end >= 0 else len(text)]
def reasons_table():
"""§13.9's codes, in order."""
return [r.split("`")[1] for r in section("13.9 Reasons").splitlines() if r.startswith("| `")]
# ------------------------------------------------------------------ §13.2
def ref(n, b):
"""The first n bytes of SHA-256 over b: a reference, and with n = 4 a message's check."""
return hashlib.sha256(b).digest()[:n]
def check_of(record):
return ref(CHECK, record)
def put_uint(v):
"""Unsigned LEB128 in its fewest bytes."""
out = bytearray()
while True:
b, v = v & 0x7F, v >> 7
if v == 0:
out.append(b)
return bytes(out)
out.append(b | 0x80)
def instant_text(s):
return (datetime.datetime(1970, 1, 1) + datetime.timedelta(seconds=s)).strftime("%Y-%m-%dT%H:%M:%SZ")
def instant_of(text):
"""The seconds of an issued_at written YYYY-MM-DDTHH:MM:SSZ, a second below 60, from 1970 to
9999; None for any other text."""
m = re.fullmatch(r"([0-9]{4})-([0-9]{2})-([0-9]{2})T([0-9]{2}):([0-9]{2}):([0-9]{2})Z", text)
if not m or not CV.is_rfc3339(text):
return None
y, mo, d, h, mi, s = (int(x) for x in m.groups())
if y < 1970 or s > 59:
return None
return calendar.timegm((y, mo, d, h, mi, s, 0, 0, 0))
def one(found, key):
"""§13.2: the one value found under a reference; values with one key are one value."""
keys = {key(x) for x in found}
if not keys:
raise Refused("compact_reference_unknown")
if len(keys) > 1 and not drops("ambiguous"):
raise Refused("compact_reference_ambiguous")
return found[0]
class Reader:
"""Reads a compact record; `kind` names it once read, so that a weakened run can drop a rule
for one kind of record alone."""
def __init__(self, b):
self.b, self.at, self.kind = b, 0, ""
def take(self, n):
if self.at + n > len(self.b):
raise Refused("compact_malformed")
s = self.b[self.at:self.at + n]
self.at += n
return s
def byte(self):
return self.take(1)[0]
def uint(self):
v = 0
for i in range(8):
b = self.byte()
v |= (b & 0x7F) << (7 * i)
if not b & 0x80:
if (i > 0 and b == 0 and not drops("minimal_" + self.kind)) \
or (v > MAX_INT and not drops("bound_" + self.kind)):
raise Refused("compact_malformed")
return v
raise Refused("compact_malformed")
def end(self):
if self.at != len(self.b) and not drops("trailing_" + self.kind):
raise Refused("compact_malformed")
# ------------------------------------------------------------------ §13.1: the context
_CONTEXTS = {}
def included(o, g):
try:
JV.inclusion(o, g)
return True
except JV.Refuse:
return False
def context_of(v, groups):
"""A context from the corpus's form: each group by name, its object receipts at the
positions listed (those that pass jwp-receipts §3 step 5), its bearer instances; a group
whose receipt fails §3 steps 1 and 2 is not held."""
memo = json.dumps([v, groups], sort_keys=True)
if memo in _CONTEXTS:
return _CONTEXTS[memo]
ctx = {"groups": [], "signers": []}
for line in v["groups"]:
e = groups[line["group"]]
try:
g = JV.read_group(e["receipt"])
JV.verify_group(g)
except JV.Refuse:
continue
objs = [o for o in (JV.read_object(e["objects"][i]) for i in line["objects"]) if included(o, g)]
link = JV.link(g)
held = next((x for x in ctx["groups"] if x["link"] == link), None)
if held is None:
ctx["groups"].append({"g": g, "link": link, "objects": objs, "bearers": list(line["bearers"])})
else:
held["objects"] += objs
held["bearers"] += line["bearers"]
ctx["signers"] = [(bytes.fromhex(s["key"]), s["signer_id"]) for s in v["signers"]]
_CONTEXTS[memo] = ctx
return ctx
def group_at(ctx, r):
return one([x for x in ctx["groups"] if ref(8, x["link"]) == r], lambda x: x["link"])
def bearer_at(ctx, grp, r):
"""Among the instances of the group the record names, not of every group held."""
held = [b for x in ctx["groups"] for b in x["bearers"]] if drops("own_group") else grp["bearers"]
return one([b for b in held if ref(4, b.encode()) == r], lambda b: b)
def signer_at(ctx, r):
return one([s for s in ctx["signers"] if ref(8, s[0]) == r], lambda s: s)
def key_at(ctx, r):
"""For a holdings record, which names no signer_id: one key under two signer_ids is one key."""
if drops("key_alone"):
return signer_at(ctx, r)[0]
return one([s[0] for s in ctx["signers"] if ref(8, s[0]) == r], lambda k: k)
def group_by_link(ctx, link):
return next((x for x in ctx["groups"] if x["link"] == link), None)
# ------------------------------------------------------------------ §13.3: a compact reconciliation
def full_bitmap(n):
b = bytearray(b"\xff" * ((n + 7) // 8))
if n % 8:
b[-1] = (0xFF << (8 - n % 8)) & 0xFF
return bytes(b)
def bit(b, i):
return b[i // 8] & (0x80 >> (i % 8)) != 0
def received_root(grp, received):
leaves = []
for i in range(grp["g"]["leaf_count"]):
if bit(received, i):
o = next((o for o in grp["objects"] if o["object_id"] == i), None)
if o is None:
if drops("objects_held"):
continue
raise Refused("compact_objects_not_held")
leaves.append(JV.leaf(o))
return BV.merkle_root(leaves)
def expand_reconciliation(r, ctx):
bad = Refused("compact_malformed")
form = r.byte()
if form & 0x03 == 3 or (form >> 3) & 0x03 == 3 or (form & 0xC0 and not drops("form_reserved")):
raise bad
bearer, whole, signer_form = BEARERS[form & 0x03], form & 0x04 != 0, (form >> 3) & 0x03
grp = group_at(ctx, r.take(8))
bearer_id = bearer_at(ctx, grp, r.take(4))
seq = r.uint()
issued = r.uint()
if issued > MAX_SECONDS and not drops("issued_bound"):
raise bad
if signer_form == 0:
key = r.take(32)
signer_id = CV.did_key(key)
elif signer_form == 1:
key = r.take(32)
raw = r.take(r.uint())
try:
signer_id = raw.decode("utf-8", errors="replace" if drops("utf8") else "strict")
except UnicodeDecodeError:
raise bad
if signer_id == CV.did_key(key) and not drops("did_key_form"):
raise bad
else:
key, signer_id = signer_at(ctx, r.take(8))
parent = r.take(32) if form & 0x20 else None
n = grp["g"]["leaf_count"]
if whole:
received = full_bitmap(n)
else:
received = r.take((n + 7) // 8)
past = any(bit(received, i) for i in range(n, 8 * len(received)))
if (past and not drops("past_bits")) or (not any(received) and not drops("no_bits")) \
or (received == full_bitmap(n) and not drops("whole_spelling")):
raise bad
sig = r.take(64)
r.end()
root = bytes.fromhex(grp["g"]["root_hash"]) if whole else received_root(grp, received)
claim = {
"kind": "claim", "class": "reconciliation", "subject_type": "receipt", "subject": grp["link"].hex(),
"seq": seq, "issued_at": instant_text(issued),
"body": {"bearer": bearer, "bearer_id": bearer_id, "leaf_count": n, "received": received.hex(),
"received_root": root.hex(), "attests": "received"},
"signer_pubkey": key.hex(), "signer_id": signer_id,
"parent_receipt_hash": parent.hex() if parent is not None else None, "sig": sig.hex(),
}
return CV.jcs(claim)
# ------------------------------------------------------------------ §13.4: a compact holdings record
def expand_holdings(r, ctx):
bad = Refused("compact_malformed")
form = r.byte()
if form == 0:
holder = r.take(32)
elif form == 1:
holder = key_at(ctx, r.take(8))
else:
raise bad
as_of, window, n = r.uint(), r.uint(), r.uint()
if n > 2 ** 32 - 1:
raise bad
out = bytearray(b"WHD1\x01" + holder + as_of.to_bytes(8, "big") + (as_of + window).to_bytes(8, "big")
+ n.to_bytes(4, "big"))
for _ in range(n):
f = r.byte()
if f & ~0x03 and not drops("form_reserved"):
raise bad
if f & 0x02:
grp = group_at(ctx, r.take(8))
group, leaf_count = grp["link"], grp["g"]["leaf_count"]
else:
group, leaf_count = r.take(32), r.uint()
out += group + leaf_count.to_bytes(8, "big")
if f & 0x01:
out.append(1)
else:
out.append(0)
out += r.take((leaf_count + 7) // 8)
out += r.take(64)
r.end()
return bytes(out)
def expand(compact, ctx):
"""(kind, record): a reconciliation's canonical text, or a holdings record's bytes."""
if len(compact) > MAX_RECORD:
raise Refused("compact_malformed")
r = Reader(compact)
kind = r.byte()
if kind == KIND_RECONCILIATION:
r.kind = "reconciliation"
return "reconciliation", expand_reconciliation(r, ctx)
if kind == KIND_HOLDINGS:
r.kind = "holdings"
return "holdings", expand_holdings(r, ctx)
raise Refused("compact_malformed")
# ------------------------------------------------------------------ §13.5: writing a record
def encode_reconciliation(text, ctx):
no = Refused("compact_not_encodable")
try:
CV.verify_claim(text)
except CV.Reject as e:
if e.step <= 2: # steps 1 and 2: it does not read
raise no
c = CV.parse_strict(text)
if c["class"] != "reconciliation" or c["subject_type"] != "receipt" or CV.jcs(c) != text:
raise no
grp = group_by_link(ctx, bytes.fromhex(c["subject"]))
issued = instant_of(c["issued_at"])
if grp is None or issued is None:
raise no
b, key = c["body"], bytes.fromhex(c["signer_pubkey"])
# The claim's own leaf_count is the group's before anything is sized by it.
if b["leaf_count"] != grp["g"]["leaf_count"]:
raise no
received = bytes.fromhex(b["received"])
whole = received == full_bitmap(grp["g"]["leaf_count"])
me = (key, c["signer_id"])
try:
by_ref = signer_at(ctx, ref(8, key)) == me and not drops("by_reference")
except Refused:
by_ref = False
signer_form = 2 if by_ref else 0 if c["signer_id"] == CV.did_key(key) else 1
form = BEARERS.index(b["bearer"]) | (signer_form << 3) | (0x04 if whole else 0) \
| (0x20 if c["parent_receipt_hash"] is not None else 0)
out = bytes([KIND_RECONCILIATION, form]) + ref(8, grp["link"]) + ref(4, b["bearer_id"].encode()) \
+ put_uint(c["seq"]) + put_uint(issued)
if signer_form == 0:
out += key
elif signer_form == 1:
sid = c["signer_id"].encode()
out += key + put_uint(len(sid)) + sid
else:
out += ref(8, key)
if c["parent_receipt_hash"] is not None:
out += bytes.fromhex(c["parent_receipt_hash"])
if not whole:
out += received
return out + bytes.fromhex(c["sig"])
def encode_holdings(b, ctx):
try:
holder, as_of, expires, entries, _ = HV.read(b)
except HV.Refused:
raise Refused("compact_not_encodable")
try:
holder_ref = key_at(ctx, ref(8, holder)) == holder and not drops("by_reference")
except Refused:
holder_ref = False
out = bytes([KIND_HOLDINGS, 1 if holder_ref else 0]) + (ref(8, holder) if holder_ref else holder)
out += put_uint(as_of) + put_uint(expires - as_of) + put_uint(len(entries))
for group, leaf_count, bits in entries:
try:
grp = group_at(ctx, ref(8, group))
by_ref = grp["link"] == group and grp["g"]["leaf_count"] == leaf_count and not drops("by_reference")
except Refused:
by_ref = False
out += bytes([(1 if bits is None else 0) | (2 if by_ref else 0)])
out += ref(8, group) if by_ref else group + put_uint(leaf_count)
if bits is not None:
out += bits
return out + b[-64:]
def encode(kind, record, ctx):
"""§13.5: the record's compact form against the context, which must expand to the record."""
out = encode_reconciliation(record, ctx) if kind == "reconciliation" else encode_holdings(record, ctx)
try:
back = expand(out, ctx)[1]
except Refused:
back = None
if back != record and not drops("roundtrip"):
raise Refused("compact_not_encodable")
return out
# ------------------------------------------------------------------ §13.6, §13.7: fragments
def message(record):
return record + check_of(record)
def cut(record, number, frame):
m = message(record)
if frame <= HEADER or len(m) > MAX_MESSAGE:
return None
per = frame - HEADER
parts = [m[i:i + per] for i in range(0, len(m), per)]
if len(parts) > 255:
return None
return [bytes([number, i, len(parts)]) + d for i, d in enumerate(parts)]
def reassemble(fragments):
"""§13.7, in its order: the compact record the fragments carry. Fragments that are the same
bytes are one fragment."""
if not drops("held_once"):
fragments = list(dict.fromkeys(fragments))
if not fragments:
raise Refused("fragment_missing")
if not drops("header") and any(len(f) <= HEADER or f[2] == 0 or f[1] >= f[2] for f in fragments):
raise Refused("fragment_malformed")
number, count = fragments[0][0], fragments[0][2]
if not drops("foreign") and any(f[0] != number or f[2] != count for f in fragments):
raise Refused("fragment_foreign")
slots = {}
for f in fragments:
if f[1] in slots and not drops("repeated"):
raise Refused("fragment_repeated")
slots[f[1]] = f[HEADER:]
if any(i not in slots for i in range(count)) and not drops("missing"):
raise Refused("fragment_missing")
m = b"".join(slots.get(i, b"") for i in range(count))
if len(m) > MAX_MESSAGE:
raise Refused("fragment_malformed")
if len(m) <= CHECK or (m[-CHECK:] != check_of(m[:-CHECK]) and not drops("check")):
raise Refused("fragment_check_failed")
return m[:-CHECK]
# ------------------------------------------------------------------ verdicts of the records
def verdict(kind, record, ctx):
"""A reconciliation through jwp-receipts §3.1 steps 1 to 4, then step 6 by bearer-receipts
§5.3 against the context's group and objects; a holdings record under holdings §4.4."""
if kind == "holdings":
try:
return "read digest=" + HV.digest(HV.read(record)).hex()
except HV.Refused as r:
return "reject reason=" + r.reason
try:
c = CV.verify_claim(record)
grp = group_by_link(ctx, bytes.fromhex(c["subject"]))
if grp is None:
return "reject step=6 reason=context_invalid"
BV.check_reconciliation(c, grp["g"], grp["objects"])
except CV.Reject as r:
return f"reject step={r.step} reason={r.reason}"
return "accept"
# ------------------------------------------------------------------ §13.8: the floors
def floors():
"""§13.8's rows: (code, name, floor)."""
rows = []
for line in section("13.8 ").splitlines():
if line.startswith("| ") and "`" in line:
cells = [c.strip() for c in line.strip("|").split("|")]
rows.append((int(cells[0].split(" ")[0]), cells[0].split("`")[1], int(cells[1].replace(",", ""))))
return rows
def measures():
"""telemetry-aggregation §3's table: code -> measure."""
out = {}
for line in open(AGGREGATION, encoding="utf-8").read().splitlines():
m = re.match(r"\| ([0-9]+) \| `([a-z_]+)` \|", line)
if m:
out[int(m.group(1))] = m.group(2)
return out
def report_lengths(upload):
"""The length of each report of a DAP upload (UploadRequest: reports, one after another)."""
r, out = TV.Rd(upload), []
while not r.done():
start = r.i
TV.read_report(r)
out.append(r.i - start)
return out
def check_floors():
fails = []
rows = floors()
if [r[0] for r in rows] != [1, 2, 3, 4]:
fails.append(f"§13.8 states the floors of measures {[r[0] for r in rows]}, not 1 to 4")
named = measures()
flows = {f["name"]: f for f in json.load(open(AGGREGATE_CORPUS, encoding="utf-8"))["flows"]}
for code, name, floor in rows:
if named.get(code) != name:
fails.append(f"§13.8 names measure {code} {name}; telemetry-aggregation names it {named.get(code)}")
lengths = set(report_lengths(bytes.fromhex(flows[name]["upload"])))
if lengths != {floor}:
fails.append(f"{name}: the reports are {sorted(lengths)} bytes, §13.8 states {floor}")
return fails
# ------------------------------------------------------------------ the corpus
def record_of(kind, v):
return v if kind == "reconciliation" else bytes.fromhex(v)
def record_value(kind, record):
return record if kind == "reconciliation" else record.hex()
def refuse(reason):
return {"outcome": "refuse", "reason": reason}
def run(kind, c, doc):
ctx = context_of(doc["contexts"][c["context"]], doc["groups"]) if "context" in c else None
if kind == "records":
record = record_of(c["kind"], c["record"])
stated = bytes.fromhex(c["compact"])
if encode(c["kind"], record, ctx) != stated:
return {"error": "written otherwise"}
if expand(stated, ctx) != (c["kind"], record):
return {"error": "expands to another record"}
return {"verdict": verdict(c["kind"], record, ctx)}
if kind == "expansions":
try:
k, record = expand(bytes.fromhex(c["compact"]), ctx)
except Refused as r:
return refuse(r.reason)
return {"outcome": "expand", "kind": k, "record": record_value(k, record), "verdict": verdict(k, record, ctx)}
if kind == "encodings":
try:
return {"outcome": "encode", "compact": encode(c["kind"], record_of(c["kind"], c["record"]), ctx).hex()}
except Refused as r:
return refuse(r.reason)
if kind == "messages":
record = bytes.fromhex(c["record"])
fragments = cut(record, c["message"], c["frame"])
if reassemble(fragments) != record:
return {"error": "its fragments do not reassemble to it"}
return {"fragments": [f.hex() for f in fragments]}
try:
return {"outcome": "reassembled", "record": reassemble([bytes.fromhex(f) for f in c["fragments"]]).hex()}
except Refused as r:
return refuse(r.reason)
KINDS = ("records", "expansions", "encodings", "messages", "reassemblies")
def strings(v, out):
if isinstance(v, str):
out.append(v)
elif isinstance(v, list):
for x in v:
strings(x, out)
elif isinstance(v, dict):
for x in v.values():
strings(x, out)
def check(doc, quiet=False):
"""The failures of a corpus document: each case's outcome, and every reason of §13.9."""
fails = []
for kind in KINDS:
for c in doc[kind]:
try:
got = run(kind, c, doc)
except (Refused, KeyError, TypeError, ValueError, IndexError, OverflowError, AttributeError,
MemoryError) as e:
got = {"error": repr(e)}
if got != c["outcome"]:
fails.append(c["name"])
if not quiet:
print(f"FAIL {c['name']}\n stated {json.dumps(c['outcome'])[:300]}\n got {json.dumps(got)[:300]}")
stated = []
for kind in KINDS:
for c in doc[kind]:
strings(c["outcome"], stated)
for code in reasons_table():
if code not in stated:
fails.append("no case gives " + code)
return fails
def mutations(doc):
"""Copies of the corpus, one defect each, every one of which must fail."""
def case(d, name):
return next(c for k in KINDS for c in d[k] if c["name"] == name)
def m_compact_byte(d):
c = case(d, "whole_did_key")
b = bytearray.fromhex(c["compact"])
b[20] ^= 0x01
c["compact"] = b.hex()
def m_record(d):
c = case(d, "partial_enrolled")
c["record"] = c["record"].replace('"seq":0', '"seq":1')
def m_verdict(d):
case(d, "labelled_label_id")["outcome"]["verdict"] = "reject step=6 reason=reconciliation_root_mismatch"
def m_holdings(d):
c = case(d, "holdings_by_reference")
c["context"] = "no_signers"
def m_context(d):
case(d, "objects_not_held")["context"] = "receiver"
def m_reason(d):
case(d, "bearer_ambiguous")["outcome"]["reason"] = "compact_reference_unknown"
def m_encodable(d):
case(d, "issued_at_fraction")["record"] = case(d, "whole_did_key")["record"]
def m_frame(d):
case(d, "whole_frame_11")["frame"] = 12
def m_number(d):
case(d, "holdings_frame_51")["message"] = 6
def m_reassembly(d):
case(d, "reordered_data_exchanged")["fragments"] = list(case(d, "in_order")["fragments"])
def m_missing_stated_whole(d):
case(d, "missing_middle")["outcome"] = {"outcome": "reassembled", "record": case(d, "whole_did_key")["compact"]}
def m_reason_gone(d):
d["expansions"].remove(case(d, "objects_not_held"))
def m_twice_refused(d):
case(d, "delivered_twice")["outcome"] = {"outcome": "refuse", "reason": "fragment_repeated"}
def m_bearer_elsewhere(d):
case(d, "bearer_of_other_groups")["context"] = "receiver"
return [m_compact_byte, m_record, m_verdict, m_holdings, m_context, m_reason, m_encodable, m_frame, m_number,
m_reassembly, m_missing_stated_whole, m_reason_gone, m_twice_refused, m_bearer_elsewhere]
WEAKENINGS = ["minimal_reconciliation", "minimal_holdings", "bound_reconciliation", "bound_holdings",
"trailing_reconciliation", "trailing_holdings", "form_reserved", "issued_bound", "utf8", "did_key_form",
"past_bits", "no_bits", "whole_spelling", "ambiguous", "own_group", "key_alone", "objects_held",
"by_reference", "roundtrip", "header", "foreign", "held_once", "repeated", "missing", "check"]
def main(argv):
global DROPPED
if len(argv) != 2:
print(__doc__)
return 2
doc = json.load(open(os.path.join(argv[1], "vectors.json"), encoding="utf-8"))
extra = sorted(set(os.listdir(argv[1])) - {"vectors.json", "README.md"})
fails = check(doc)
if extra:
fails.append(f"files the corpus does not hold: {extra}")
# The bounds no vector carries (65 KB of hexadecimal): checked here.
try:
expand(bytes([KIND_HOLDINGS]) * (MAX_RECORD + 1), {"groups": [], "signers": []})
fails.append("a compact record past 65,531 bytes was read")
except Refused as r:
if r.reason != "compact_malformed":
fails.append("a compact record past the bound was refused " + r.reason)
big = [bytes([7, i, 255]) + bytes(258) for i in range(255)]
try:
reassemble(big)
fails.append("a message past 65,535 bytes was reassembled")
except Refused as r:
if r.reason != "fragment_malformed":
fails.append("a message past the bound was refused " + r.reason)
if cut(bytes(MAX_RECORD), 7, 1500) is None or cut(bytes(MAX_RECORD + 1), 7, 1500) is not None:
fails.append("a message is cut up to 65,535 bytes and no further")
fails += check_floors()
print(f"{sum(len(doc[k]) for k in KINDS)} cases ({', '.join(f'{len(doc[k])} {k}' for k in KINDS)}): "
f"{len(fails)} failure(s)")
caught = 0
for m in mutations(doc):
d = copy.deepcopy(doc)
m(d)
if check(d, quiet=True):
caught += 1
else:
fails.append(f"mutation {m.__name__} was not caught")
print(f"{caught} of {len(mutations(doc))} mutated copies fail")
weak = 0
for w in WEAKENINGS:
DROPPED = w
if check(doc, quiet=True):
weak += 1
else:
fails.append(f"a verifier without its {w} check passes the corpus")
DROPPED = None
print(f"{weak} of {len(WEAKENINGS)} weakened verifiers are caught")
for f in fails:
print("FAIL", f)
return 1 if fails else 0
if __name__ == "__main__":
sys.exit(main(sys.argv))

View file

@ -1414,6 +1414,11 @@ name = "wai_bearer_vectors"
path = "src/bin/wai_bearer_vectors.rs"
required-features = ["bearer_receipts"]
[[bin]]
name = "wai_compact_return_vectors"
path = "src/bin/wai_compact_return_vectors.rs"
required-features = ["bearer_receipts", "holdings"]
[[bin]]
name = "wai_carousel_vectors"
path = "src/bin/wai_carousel_vectors.rs"
@ -2491,6 +2496,11 @@ name = "bearer_receipts"
path = "tests/bearer_receipts.rs"
required-features = ["bearer_receipts"]
[[test]]
name = "compact_return"
path = "tests/compact_return.rs"
required-features = ["bearer_receipts", "holdings", "telemetry_aggregate"]
[[test]]
name = "carousel"
path = "tests/carousel.rs"

View file

@ -0,0 +1,75 @@
//! Conformance corpus for the compact return profile (`live-conformance/compact-return/`,
//! extensions/bearer-receipts.md §13.10).
//!
//! wai_compact_return_vectors gen <corpus-dir> write vectors.json, from fixed seeds
//! wai_compact_return_vectors verify <corpus-dir> every file regenerates byte for byte, no
//! other file but README.md is there, and
//! every case gets the outcome it states
//!
//! The generator and this verifier are one implementation. The corpus's second
//! implementation is `live-conformance/compact_return_verify.py`, written from the extension's
//! text and sharing no code with this binary.
use std::collections::BTreeMap;
use std::path::Path;
use wai::compact_return::corpus;
fn main() {
let a: Vec<String> = std::env::args().collect();
match (a.get(1).map(String::as_str), a.get(2)) {
(Some("gen"), Some(d)) => {
if let Err(e) = std::fs::create_dir_all(d) {
eprintln!("create {d}: {e}");
std::process::exit(1);
}
for (name, body) in corpus::generate() {
let p = Path::new(d).join(&name);
if let Err(e) = std::fs::write(&p, body) {
eprintln!("write {}: {e}", p.display());
std::process::exit(1);
}
}
println!("wrote {d}");
}
(Some("verify"), Some(d)) => std::process::exit(verify(Path::new(d))),
_ => {
eprintln!("usage: wai_compact_return_vectors gen|verify <corpus-dir>");
std::process::exit(2);
}
}
}
fn verify(dir: &Path) -> i32 {
let mut files = BTreeMap::new();
let entries = match std::fs::read_dir(dir) {
Ok(e) => e,
Err(e) => {
eprintln!("read {}: {e}", dir.display());
return 1;
}
};
for e in entries.flatten() {
let name = e.file_name().to_string_lossy().into_owned();
if name == "README.md" {
continue;
}
match std::fs::read_to_string(e.path()) {
Ok(s) => {
files.insert(name, s);
}
Err(err) => {
eprintln!("read {name}: {err}");
return 1;
}
}
}
let generated: BTreeMap<String, String> = corpus::generate().into_iter().collect();
let same = files == generated;
println!("{} every file regenerates byte for byte from its seeds, and no other file is there", if same { "PASS" } else { "FAIL" });
let fails = corpus::verify(&files);
for f in &fails {
println!("FAIL {f}");
}
println!("{} every case gets the outcome the corpus states", if fails.is_empty() { "PASS" } else { "FAIL" });
(!same || !fails.is_empty()) as i32
}

View file

@ -0,0 +1,682 @@
//! The compact return profile — `extensions/bearer-receipts.md` §13: a reconciliation (§5) and a
//! holdings record (`extensions/holdings.md` §4) in a binary form for return paths whose messages
//! hold few bytes, carried in fragments with a check over the whole message.
//!
//! A compact record names by short references (§13.2) what its receiver already holds — the
//! group, the bearer instance, the signer — in a [`Context`]. [`expand`] turns it, against that
//! context, into the record's own bytes: a reconciliation claim's canonical form, a holdings
//! record's `WHD1`. [`encode`] is its inverse for a record that has a compact form (§13.5), and
//! refuses one that has none. The expansion is verified by the record's own text — `claim_receipt`
//! and bearer-receipts §5.3 for a reconciliation, [`crate::holdings`] for a holdings record — and
//! never here. [`cut`] and [`reassemble`] carry a message in fragments (§13.6, §13.7).
//!
//! # Rules enforced here rather than only documented
//!
//! - A reference resolves to one value or to none: two values with one reference refuse it
//! (`compact_reference_ambiguous`), never resolved by choosing.
//! - A compact record has one reading and each form one spelling: a uint in its fewest bytes, a
//! signer written inline with its own `did:key` is form 0 and never form 1, and a partial
//! bitmap that sets every bit is the whole form's, so a reader refuses the other spelling.
//! - [`encode`] expands what it wrote against the same context and refuses unless that is the
//! record it was given, byte for byte: what it returns round-trips.
//! - The check over a message is not a signature: whatever a message expands to is verified by
//! its own signature, and the compact form carries no statement of its own.
//! - Fragments that are the same bytes are one fragment, so a fragment the path delivers twice is
//! held once; two that differ at one index refuse the message (`fragment_repeated`).
//! - [`encode`] sizes nothing by a reconciliation's own `leaf_count` before it has found it to be
//! the group's: a record received from a neighbour states it, and steps 1 and 2 bound it only by
//! 2^53 − 1.
use std::collections::BTreeSet;
use sha2::{Digest, Sha256};
use crate::claim_receipt::bearer::ATTESTS_RECEIVED;
use crate::claim_receipt::{rfc3339_seconds, rfc3339_utc, Bearer, ClaimBody, ClaimReceipt, ReconciliationClaim, SubjectType};
use crate::holdings::{self, Holdings};
use crate::jwp_delivery::{root as merkle_root, GroupReceipt, ObjectReceipt};
use crate::signer_id::did_key;
pub mod corpus;
/// The codes bearer-receipts §13.9 lists, in its order.
pub const CODES: &[&str] = &[
MALFORMED,
REFERENCE_UNKNOWN,
REFERENCE_AMBIGUOUS,
OBJECTS_NOT_HELD,
NOT_ENCODABLE,
FRAGMENT_MALFORMED,
FRAGMENT_FOREIGN,
FRAGMENT_REPEATED,
FRAGMENT_MISSING,
FRAGMENT_CHECK_FAILED,
];
pub const MALFORMED: &str = "compact_malformed";
pub const REFERENCE_UNKNOWN: &str = "compact_reference_unknown";
pub const REFERENCE_AMBIGUOUS: &str = "compact_reference_ambiguous";
pub const OBJECTS_NOT_HELD: &str = "compact_objects_not_held";
pub const NOT_ENCODABLE: &str = "compact_not_encodable";
pub const FRAGMENT_MALFORMED: &str = "fragment_malformed";
pub const FRAGMENT_FOREIGN: &str = "fragment_foreign";
pub const FRAGMENT_REPEATED: &str = "fragment_repeated";
pub const FRAGMENT_MISSING: &str = "fragment_missing";
pub const FRAGMENT_CHECK_FAILED: &str = "fragment_check_failed";
/// §13.3: a compact reconciliation's `kind`.
pub const KIND_RECONCILIATION: u8 = 0x01;
/// §13.4: a compact holdings record's `kind`.
pub const KIND_HOLDINGS: u8 = 0x02;
/// §13.6: the bytes of a fragment before its data.
pub const HEADER: usize = 3;
/// §13.1: the bytes of a message's check.
pub const CHECK: usize = 4;
/// §13.6: the most bytes a message has.
pub const MAX_MESSAGE: usize = 65_535;
/// §13.3, §13.4: the most bytes a compact record has, so that its message is at most
/// [`MAX_MESSAGE`].
pub const MAX_RECORD: usize = MAX_MESSAGE - CHECK;
/// §13.2: the greatest instant, 9999-12-31T23:59:59Z.
pub const MAX_SECONDS: u64 = 253_402_300_799;
/// §13.2: the greatest uint, 2^53 − 1.
pub const MAX_INT: u64 = (1 << 53) - 1;
// ───────────────────────────── §13.2 ─────────────────────────────
/// The first `N` bytes of SHA-256 over `bytes`: a reference (§13.2), and with `N` = 4 a
/// message's check (§13.1).
pub fn reference<const N: usize>(bytes: &[u8]) -> [u8; N] {
Sha256::digest(bytes)[..N].try_into().expect("N is at most 32")
}
/// A group's reference: 8 bytes over its link hash.
pub fn group_ref(link_hash: &[u8; 32]) -> [u8; 8] {
reference(link_hash)
}
/// A bearer instance's reference: 4 bytes over its `bearer_id` in UTF-8.
pub fn bearer_ref(bearer_id: &str) -> [u8; 4] {
reference(bearer_id.as_bytes())
}
/// A signer's reference: 8 bytes over its key.
pub fn signer_ref(key: &[u8; 32]) -> [u8; 8] {
reference(key)
}
/// A message's check (§13.1): the first 4 bytes of SHA-256 over its compact record.
pub fn check(record: &[u8]) -> [u8; CHECK] {
reference(record)
}
/// Write `v` as a uint (§13.2): unsigned LEB128 in its fewest bytes.
pub fn put_uint(out: &mut Vec<u8>, mut v: u64) {
loop {
let b = (v & 0x7f) as u8;
v >>= 7;
if v == 0 {
out.push(b);
return;
}
out.push(b | 0x80);
}
}
/// The instant of an `issued_at` (§13.2): seconds since 1970-01-01T00:00:00Z, for a text written
/// `YYYY-MM-DDTHH:MM:SSZ` with a second below 60 and at most [`MAX_SECONDS`]; `None` for any other
/// text, which has no instant.
pub fn instant_of(text: &str) -> Option<u64> {
let s = u64::try_from(rfc3339_seconds(text)?).ok()?;
(s <= MAX_SECONDS && rfc3339_utc(s) == text).then_some(s)
}
struct Reader<'a> {
b: &'a [u8],
at: usize,
}
impl<'a> Reader<'a> {
fn take(&mut self, n: usize) -> Result<&'a [u8], &'static str> {
let s = self.at.checked_add(n).and_then(|e| self.b.get(self.at..e)).ok_or(MALFORMED)?;
self.at += n;
Ok(s)
}
fn byte(&mut self) -> Result<u8, &'static str> {
Ok(self.take(1)?[0])
}
fn array<const N: usize>(&mut self) -> Result<[u8; N], &'static str> {
Ok(self.take(N)?.try_into().expect("N bytes"))
}
/// §13.2: a uint in its fewest bytes, at most 2^53 − 1, so at most 8 bytes.
fn uint(&mut self) -> Result<u64, &'static str> {
let mut v = 0u64;
for i in 0..8 {
let b = self.byte()?;
v |= u64::from(b & 0x7f) << (7 * i);
if b & 0x80 == 0 {
return if (i > 0 && b == 0) || v > MAX_INT { Err(MALFORMED) } else { Ok(v) };
}
}
Err(MALFORMED)
}
fn end(&self) -> Result<(), &'static str> {
if self.at == self.b.len() {
Ok(())
} else {
Err(MALFORMED)
}
}
}
// ───────────────────────────── §13.1: the context ─────────────────────────────
/// One group of a receiver's context (§13.1).
#[derive(Clone, Debug)]
pub struct ContextGroup {
pub receipt: GroupReceipt,
/// Its link hash, computed once.
pub link: [u8; 32],
/// The object receipts held under it that pass jwp-receipts §3 step 5 against it.
pub objects: Vec<ObjectReceipt>,
/// The `bearer_id` of each instance of the group the receiver holds a declaration of.
pub bearers: Vec<String>,
}
/// A key a receiver holds, with its `signer_id` (§13.1).
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub struct Signer {
pub key: [u8; 32],
pub signer_id: String,
}
/// What a receiver holds that a compact record names (§13.1).
#[derive(Clone, Debug, Default)]
pub struct Context {
pub groups: Vec<ContextGroup>,
pub signers: Vec<Signer>,
}
/// Of the values found under one reference, the one value they are, or why not (§13.2):
/// values with one key are one value.
fn one<T, K: Ord>(found: Vec<T>, key: impl Fn(&T) -> K) -> Result<T, &'static str> {
let distinct: BTreeSet<K> = found.iter().map(key).collect();
match distinct.len() {
0 => Err(REFERENCE_UNKNOWN),
1 => Ok(found.into_iter().next().expect("one value")),
_ => Err(REFERENCE_AMBIGUOUS),
}
}
impl Context {
/// Add a group, with the object receipts and bearer instances held under it. A group whose
/// receipt fails jwp-receipts §3 steps 1 and 2 is not held, nor is an object receipt that
/// fails step 5 against it; a group already held gains the objects and instances.
pub fn add_group(&mut self, receipt: GroupReceipt, objects: Vec<ObjectReceipt>, bearers: Vec<String>) {
if receipt.verify().is_err() {
return;
}
let objects: Vec<ObjectReceipt> = objects.into_iter().filter(|o| o.verify_inclusion(&receipt).is_ok()).collect();
let link = receipt.link_hash();
match self.groups.iter_mut().find(|g| g.link == link) {
Some(g) => {
g.objects.extend(objects);
g.bearers.extend(bearers);
}
None => self.groups.push(ContextGroup { receipt, link, objects, bearers }),
}
}
/// The group a reference resolves to.
pub fn group(&self, r: &[u8; 8]) -> Result<&ContextGroup, &'static str> {
one(self.groups.iter().filter(|g| group_ref(&g.link) == *r).collect(), |g| g.link)
}
/// The group whose link hash is `link`, if the context holds it.
pub fn group_by_link(&self, link: &[u8; 32]) -> Option<&ContextGroup> {
self.groups.iter().find(|g| g.link == *link)
}
/// The signer, key and `signer_id`, a reference resolves to: for a reconciliation.
pub fn signer(&self, r: &[u8; 8]) -> Result<&Signer, &'static str> {
one(self.signers.iter().filter(|s| signer_ref(&s.key) == *r).collect(), |s| (s.key, s.signer_id.clone()))
}
/// The key a reference resolves to: for a holdings record, which names no `signer_id`.
pub fn key(&self, r: &[u8; 8]) -> Result<[u8; 32], &'static str> {
one(self.signers.iter().filter(|s| signer_ref(&s.key) == *r).map(|s| s.key).collect(), |k| *k)
}
}
impl ContextGroup {
/// The bearer instance of this group a reference resolves to.
pub fn bearer(&self, r: &[u8; 4]) -> Result<&str, &'static str> {
one(self.bearers.iter().filter(|b| bearer_ref(b) == *r).collect(), |b| b.to_string()).map(|b| b.as_str())
}
}
// ───────────────────────────── records ─────────────────────────────
/// A record in its own form: what a compact record expands to.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum Record {
/// A reconciliation claim's canonical form (jwp-receipts §2.4.3).
Reconciliation(String),
/// A holdings record's bytes (holdings §4).
Holdings(Vec<u8>),
}
impl Record {
pub fn bytes(&self) -> &[u8] {
match self {
Record::Reconciliation(s) => s.as_bytes(),
Record::Holdings(b) => b,
}
}
}
/// §13.3, §13.4: a compact record, expanded against `ctx` into its record's own bytes.
pub fn expand(compact: &[u8], ctx: &Context) -> Result<Record, &'static str> {
if compact.len() > MAX_RECORD {
return Err(MALFORMED);
}
let mut r = Reader { b: compact, at: 0 };
match r.byte()? {
KIND_RECONCILIATION => expand_reconciliation(&mut r, ctx).map(Record::Reconciliation),
KIND_HOLDINGS => expand_holdings(&mut r, ctx).map(Record::Holdings),
_ => Err(MALFORMED),
}
}
/// §13.5: a record's compact form against `ctx`, or `compact_not_encodable`.
pub fn encode(record: &Record, ctx: &Context) -> Result<Vec<u8>, &'static str> {
let out = match record {
Record::Reconciliation(text) => encode_reconciliation(text, ctx)?,
Record::Holdings(bytes) => encode_holdings(bytes, ctx)?,
};
// What it wrote is the record's only if it expands to it, here and now.
match expand(&out, ctx) {
Ok(back) if back == *record => Ok(out),
_ => Err(NOT_ENCODABLE),
}
}
/// The bitmap of `leaf_count` objects that holds every one.
fn full_bitmap(leaf_count: u64) -> Vec<u8> {
let mut b = vec![0xffu8; leaf_count.div_ceil(8) as usize];
let used = (leaf_count % 8) as u32;
if used != 0 {
*b.last_mut().expect("a byte") = 0xffu8 << (8 - used);
}
b
}
/// §13.3: a partial `received`'s root, over the leaves of the object receipts the context holds
/// at its set bits, in increasing order.
fn received_root(g: &ContextGroup, received: &[u8]) -> Result<[u8; 32], &'static str> {
let mut leaves = Vec::new();
for i in 0..g.receipt.leaf_count {
if received[(i / 8) as usize] & (0x80 >> (i % 8)) != 0 {
let o = g.objects.iter().find(|o| o.object_id == i).ok_or(OBJECTS_NOT_HELD)?;
leaves.push(o.leaf());
}
}
Ok(merkle_root(&leaves))
}
fn expand_reconciliation(r: &mut Reader, ctx: &Context) -> Result<String, &'static str> {
let form = r.byte()?;
let bearer = match form & 0x03 {
0 => Bearer::Bidirectional,
1 => Bearer::Multicast,
2 => Bearer::Unidirectional,
_ => return Err(MALFORMED),
};
let whole = form & 0x04 != 0;
let signer_form = (form >> 3) & 0x03;
if form & 0xc0 != 0 || signer_form == 3 {
return Err(MALFORMED);
}
let g = ctx.group(&r.array()?)?;
let bearer_id = g.bearer(&r.array()?)?.to_string();
let seq = r.uint()?;
let issued_at = r.uint()?;
if issued_at > MAX_SECONDS {
return Err(MALFORMED);
}
let (signer_pubkey, signer_id) = match signer_form {
0 => {
let key: [u8; 32] = r.array()?;
(key, did_key(&key))
}
1 => {
let key: [u8; 32] = r.array()?;
let n = usize::try_from(r.uint()?).map_err(|_| MALFORMED)?;
let id = std::str::from_utf8(r.take(n)?).map_err(|_| MALFORMED)?.to_string();
if id == did_key(&key) {
return Err(MALFORMED);
}
(key, id)
}
_ => {
let s = ctx.signer(&r.array()?)?;
(s.key, s.signer_id.clone())
}
};
let parent_receipt_hash = if form & 0x20 != 0 { Some(r.array::<32>()?) } else { None };
let leaf_count = g.receipt.leaf_count;
let received = if whole {
full_bitmap(leaf_count)
} else {
let b = r.take(leaf_count.div_ceil(8) as usize)?.to_vec();
let used = (leaf_count % 8) as u32;
let past = used != 0 && b.last().is_some_and(|x| x & (0xffu8 >> used) != 0);
if past || b.iter().all(|x| *x == 0) || b == full_bitmap(leaf_count) {
return Err(MALFORMED);
}
b
};
let sig: [u8; 64] = r.array()?;
r.end()?;
let root = if whole { g.receipt.root_hash } else { received_root(g, &received)? };
let claim = ClaimReceipt {
subject_type: SubjectType::Receipt,
subject: g.link,
seq,
issued_at: rfc3339_utc(issued_at),
body: ClaimBody::Reconciliation(ReconciliationClaim {
bearer,
bearer_id,
leaf_count,
received,
received_root: root,
attests: ATTESTS_RECEIVED.into(),
}),
signer_pubkey,
signer_id,
parent_receipt_hash,
sig,
hybrid: None,
};
Ok(claim.to_json())
}
fn expand_holdings(r: &mut Reader, ctx: &Context) -> Result<Vec<u8>, &'static str> {
let holder: [u8; 32] = match r.byte()? {
0 => r.array()?,
1 => ctx.key(&r.array()?)?,
_ => return Err(MALFORMED),
};
let as_of_ms = r.uint()?;
let window_ms = r.uint()?;
let n = u32::try_from(r.uint()?).map_err(|_| MALFORMED)?;
let mut out = Vec::new();
out.extend_from_slice(holdings::MAGIC);
out.push(1);
out.extend_from_slice(&holder);
out.extend_from_slice(&as_of_ms.to_be_bytes());
out.extend_from_slice(&(as_of_ms + window_ms).to_be_bytes());
out.extend_from_slice(&n.to_be_bytes());
for _ in 0..n {
let form = r.byte()?;
if form & !0x03 != 0 {
return Err(MALFORMED);
}
let (group, leaf_count) = if form & 0x02 != 0 {
let g = ctx.group(&r.array()?)?;
(g.link, g.receipt.leaf_count)
} else {
(r.array::<32>()?, r.uint()?)
};
out.extend_from_slice(&group);
out.extend_from_slice(&leaf_count.to_be_bytes());
if form & 0x01 != 0 {
out.push(1);
} else {
out.push(0);
out.extend_from_slice(r.take(leaf_count.div_ceil(8) as usize)?);
}
}
out.extend_from_slice(&r.array::<64>()?);
r.end()?;
Ok(out)
}
fn encode_reconciliation(text: &str, ctx: &Context) -> Result<Vec<u8>, &'static str> {
let no = Err(NOT_ENCODABLE);
let Ok(c) = ClaimReceipt::from_json(text) else { return no };
let ClaimBody::Reconciliation(b) = &c.body else { return no };
if c.hybrid.is_some() || c.subject_type != SubjectType::Receipt || c.to_json() != text {
return no;
}
let Some(g) = ctx.group_by_link(&c.subject) else { return no };
let Some(issued_at) = instant_of(&c.issued_at) else { return no };
// The claim's own leaf_count is the group's before anything is sized by it.
if b.leaf_count != g.receipt.leaf_count {
return no;
}
let whole = b.received == full_bitmap(g.receipt.leaf_count);
let me = Signer { key: c.signer_pubkey, signer_id: c.signer_id.clone() };
let signer_form = if ctx.signer(&signer_ref(&me.key)) == Ok(&me) {
2
} else if c.signer_id == did_key(&c.signer_pubkey) {
0
} else {
1
};
let bearer: u8 = match b.bearer {
Bearer::Bidirectional => 0,
Bearer::Multicast => 1,
Bearer::Unidirectional => 2,
};
let mut form = bearer | (signer_form << 3);
if whole {
form |= 0x04;
}
if c.parent_receipt_hash.is_some() {
form |= 0x20;
}
let mut out = vec![KIND_RECONCILIATION, form];
out.extend_from_slice(&group_ref(&g.link));
out.extend_from_slice(&bearer_ref(&b.bearer_id));
put_uint(&mut out, c.seq);
put_uint(&mut out, issued_at);
match signer_form {
0 => out.extend_from_slice(&c.signer_pubkey),
1 => {
out.extend_from_slice(&c.signer_pubkey);
put_uint(&mut out, c.signer_id.len() as u64);
out.extend_from_slice(c.signer_id.as_bytes());
}
_ => out.extend_from_slice(&signer_ref(&c.signer_pubkey)),
}
if let Some(p) = &c.parent_receipt_hash {
out.extend_from_slice(p);
}
if !whole {
out.extend_from_slice(&b.received);
}
out.extend_from_slice(&c.sig);
Ok(out)
}
fn encode_holdings(bytes: &[u8], ctx: &Context) -> Result<Vec<u8>, &'static str> {
let h = Holdings::read(bytes).map_err(|_| NOT_ENCODABLE)?;
let mut out = vec![KIND_HOLDINGS];
if ctx.key(&signer_ref(&h.holder)) == Ok(h.holder) {
out.push(1);
out.extend_from_slice(&signer_ref(&h.holder));
} else {
out.push(0);
out.extend_from_slice(&h.holder);
}
put_uint(&mut out, h.as_of_ms);
put_uint(&mut out, h.expires_ms - h.as_of_ms);
put_uint(&mut out, h.entries.len() as u64);
for e in &h.entries {
let by_ref = ctx.group(&group_ref(&e.group)).is_ok_and(|g| g.link == e.group && g.receipt.leaf_count == e.leaf_count);
let whole = e.held == holdings::Held::Whole;
out.push(u8::from(whole) | if by_ref { 0x02 } else { 0 });
if by_ref {
out.extend_from_slice(&group_ref(&e.group));
} else {
out.extend_from_slice(&e.group);
put_uint(&mut out, e.leaf_count);
}
if let holdings::Held::Bitmap(bits) = &e.held {
out.extend_from_slice(bits);
}
}
out.extend_from_slice(&h.sig);
Ok(out)
}
// ───────────────────────────── §13.6, §13.7: fragments ─────────────────────────────
/// A compact record followed by its check (§13.1).
pub fn message(record: &[u8]) -> Vec<u8> {
[record, &check(record)[..]].concat()
}
/// §13.6: a compact record's message cut, in order, into fragments of `frame` bytes, the last
/// holding what remains, under the sender's number `number`. `None` when the path cannot carry
/// it: a frame below 4 bytes, a message over [`MAX_MESSAGE`], or one that needs more than 255
/// fragments.
pub fn cut(record: &[u8], number: u8, frame: usize) -> Option<Vec<Vec<u8>>> {
let m = message(record);
if frame <= HEADER || m.len() > MAX_MESSAGE {
return None;
}
let chunks: Vec<&[u8]> = m.chunks(frame - HEADER).collect();
let count = u8::try_from(chunks.len()).ok()?;
Some(chunks.iter().enumerate().map(|(i, d)| [&[number, i as u8, count][..], d].concat()).collect())
}
/// §13.7: the compact record a message's fragments, in any order, carry; or the step that
/// refuses them.
pub fn reassemble(fragments: &[Vec<u8>]) -> Result<Vec<u8>, &'static str> {
let first = fragments.first().ok_or(FRAGMENT_MISSING)?;
if fragments.iter().any(|f| f.len() <= HEADER || f[2] == 0 || f[1] >= f[2]) {
return Err(FRAGMENT_MALFORMED);
}
if fragments.iter().any(|f| f[0] != first[0] || f[2] != first[2]) {
return Err(FRAGMENT_FOREIGN);
}
// Past step 2 every fragment has one number and one count, so two at one index with the same
// data are the same bytes: one fragment.
let mut slots: Vec<Option<&[u8]>> = vec![None; usize::from(first[2])];
for f in fragments {
let slot = &mut slots[usize::from(f[1])];
match slot {
Some(held) if *held == &f[HEADER..] => {}
Some(_) => return Err(FRAGMENT_REPEATED),
None => *slot = Some(&f[HEADER..]),
}
}
let parts: Vec<&[u8]> = slots.into_iter().collect::<Option<_>>().ok_or(FRAGMENT_MISSING)?;
let m = parts.concat();
if m.len() > MAX_MESSAGE {
return Err(FRAGMENT_MALFORMED);
}
if m.len() <= CHECK || m[m.len() - CHECK..] != check(&m[..m.len() - CHECK]) {
return Err(FRAGMENT_CHECK_FAILED);
}
Ok(m[..m.len() - CHECK].to_vec())
}
#[cfg(test)]
mod tests {
use super::*;
fn read_uint(b: &[u8]) -> Result<u64, &'static str> {
let mut r = Reader { b, at: 0 };
let v = r.uint()?;
r.end().map(|()| v)
}
#[test]
fn a_uint_has_one_spelling() {
for v in [0, 1, 127, 128, 300, 16_383, 16_384, 1_793_692_800, MAX_INT] {
let mut b = Vec::new();
put_uint(&mut b, v);
assert_eq!(read_uint(&b), Ok(v), "{v}");
}
assert_eq!(read_uint(&[0x80, 0x00]), Err(MALFORMED), "0 in two bytes");
assert_eq!(read_uint(&[0xff, 0x00]), Err(MALFORMED), "127 in two bytes");
assert_eq!(read_uint(&[0xff; 7].iter().copied().chain([0x0f]).collect::<Vec<_>>()), Ok(MAX_INT));
assert_eq!(read_uint(&[0xff; 7].iter().copied().chain([0x10]).collect::<Vec<_>>()), Err(MALFORMED), "2^53");
assert_eq!(read_uint(&[0x80; 8]), Err(MALFORMED), "a ninth byte");
assert_eq!(read_uint(&[0x80]), Err(MALFORMED), "ends inside");
}
#[test]
fn an_instant_is_one_spelling_of_one_second() {
assert_eq!(instant_of("1970-01-01T00:00:00Z"), Some(0));
assert_eq!(instant_of("2026-10-22T08:00:00Z"), Some(1_792_656_000));
assert_eq!(instant_of("9999-12-31T23:59:59Z"), Some(MAX_SECONDS));
for other in ["2026-10-22T08:00:00.0Z", "2026-10-22T08:00:00+00:00", "2026-10-22t08:00:00z", "2026-10-22T08:00:60Z", "1969-12-31T23:59:59Z"] {
assert_eq!(instant_of(other), None, "{other}");
}
assert_eq!(rfc3339_utc(MAX_SECONDS), "9999-12-31T23:59:59Z");
}
#[test]
fn the_full_bitmap_leaves_the_bits_past_the_group_clear() {
assert_eq!(full_bitmap(1), vec![0x80]);
assert_eq!(full_bitmap(8), vec![0xff]);
assert_eq!(full_bitmap(9), vec![0xff, 0x80]);
assert_eq!(full_bitmap(20), vec![0xff, 0xff, 0xf0]);
}
#[test]
fn a_message_reassembles_in_any_order_and_each_defect_refuses() {
let record: Vec<u8> = (0u8..40).collect();
let f = cut(&record, 7, 11).expect("a frame of 11");
assert_eq!(f.len(), 6, "44 bytes in pieces of 8");
assert!(f.iter().all(|x| x.len() <= 11));
assert_eq!(reassemble(&f), Ok(record.clone()));
let mut back = f.clone();
back.reverse();
assert_eq!(reassemble(&back), Ok(record.clone()), "arrival order is free");
assert_eq!(reassemble(&f[1..]), Err(FRAGMENT_MISSING));
assert_eq!(reassemble(&[]), Err(FRAGMENT_MISSING));
assert_eq!(reassemble(&[f.clone(), vec![f[2].clone()]].concat()), Ok(record.clone()), "a fragment twice, byte for byte, is one");
let mut other_data = f[2].clone();
other_data[HEADER] ^= 0x01;
assert_eq!(reassemble(&[f.clone(), vec![other_data]].concat()), Err(FRAGMENT_REPEATED));
let mut swapped = f.clone();
let (a, b) = (swapped[1][HEADER..].to_vec(), swapped[2][HEADER..].to_vec());
swapped[1].truncate(HEADER);
swapped[1].extend_from_slice(&b);
swapped[2].truncate(HEADER);
swapped[2].extend_from_slice(&a);
assert_eq!(reassemble(&swapped), Err(FRAGMENT_CHECK_FAILED), "two fragments' data exchanged");
let other = cut(&[9u8; 40], 8, 11).expect("another");
let mut foreign = f.clone();
foreign[3] = other[3].clone();
assert_eq!(reassemble(&foreign), Err(FRAGMENT_FOREIGN));
let same_number = cut(&[9u8; 40], 7, 11).expect("another under the same number");
let mut mixed = f.clone();
mixed[3] = same_number[3].clone();
assert_eq!(reassemble(&mixed), Err(FRAGMENT_CHECK_FAILED));
assert_eq!(reassemble(&[vec![7, 0, 1]]), Err(FRAGMENT_MALFORMED), "no data");
assert_eq!(reassemble(&[vec![7, 1, 1, 0]]), Err(FRAGMENT_MALFORMED), "index past count");
assert_eq!(reassemble(&[vec![7, 0, 0, 0]]), Err(FRAGMENT_MALFORMED), "count 0");
assert_eq!(reassemble(&[[&[7u8, 0, 1][..], &check(&[])].concat()]), Err(FRAGMENT_CHECK_FAILED), "a check with no record");
assert!(cut(&record, 7, 3).is_none(), "a frame of the header alone");
assert!(cut(&vec![0u8; 300], 7, 4).is_none(), "more than 255 fragments");
// §13.6's bound: 255 fragments of 258 bytes of data hold more than 65,535 bytes.
let big: Vec<Vec<u8>> = (0..255u8).map(|i| [&[7u8, i, 255][..], &[0u8; 258][..]].concat()).collect();
assert_eq!(reassemble(&big), Err(FRAGMENT_MALFORMED));
assert!(cut(&vec![0u8; MAX_RECORD], 7, 1500).is_some());
assert!(cut(&vec![0u8; MAX_RECORD + 1], 7, 1500).is_none());
}
#[test]
fn a_record_past_the_bound_is_refused_before_it_is_read() {
let ctx = Context::default();
assert_eq!(expand(&vec![KIND_HOLDINGS; MAX_RECORD + 1], &ctx), Err(MALFORMED));
assert_eq!(expand(&[], &ctx), Err(MALFORMED));
assert_eq!(expand(&[0x03], &ctx), Err(MALFORMED));
}
}

View file

@ -0,0 +1,866 @@
//! The conformance corpus of extensions/bearer-receipts.md §13.10: `live-conformance/compact-return/`.
//!
//! [`generate`] writes `vectors.json` from fixed seeds: every key is an Ed25519 key from a fixed
//! 32-byte seed, every time a constant, every envelope is written here, and Ed25519 signing is
//! deterministic (RFC 8032), so the file is the same bytes on every machine. Each case states the
//! outcome it was written to have, declared here and never copied from [`super::expand`],
//! [`super::encode`] or [`super::reassemble`]; [`verify`] requires the module to reach every
//! outcome, every round trip to hold both ways, every cut to reassemble, and the corpus to give
//! every reason of §13.9. `live-conformance/compact_return_verify.py` checks the same file from
//! the extension's text, sharing no code with this module.
use super::*;
use crate::claim_receipt::{hex, unhex, Rejection, TransmissionClaim};
use crate::container::{Conditioning, Manifest, ModelRequirement, Wai};
use crate::energy_class::{CounterFiltering, EnergyProvenance, Uncertainty};
use crate::holdings::Entry;
use crate::jwp_delivery::{GroupBuilder, ObjectLabel};
use ed25519_dalek::SigningKey;
use serde_json::{json, Map, Value};
use std::collections::BTreeMap;
const PUBLISHER: u8 = 0x11;
const SINK_A: u8 = 0x41;
const SINK_B: u8 = 0x42;
const SINK_C: u8 = 0x43;
const SINK_D: u8 = 0x44;
const HOLDER: u8 = 0x51;
const NEIGHBOUR: u8 = 0x52;
const D1: &str = "did:web:publisher.example#d1";
/// Sink B signs under an origin that speaks for it, and has enrolled its key with the receiver.
const SINK_B_ID: &str = "did:web:sinks.example#b";
/// The same key under a second identifier, for an ambiguous reference.
const SINK_B_OTHER_ID: &str = "did:web:sinks.example#b2";
/// Sink C signs under a label, a `signer_id` of neither form of jwp-receipts §7.1.
const SINK_C_ID: &str = "urn:example:wai-sink:c";
/// The holder's key, as the receiver holds it.
const HOLDER_ID: &str = "did:web:holders.example#h1";
/// The same key under a second identifier: one key still, for a holdings record.
const HOLDER_OTHER_ID: &str = "did:web:holders.example#h2";
const MCAST: &str = "urn:example:wai-bearer:multicast-1";
const BCAST: &str = "urn:example:wai-bearer:broadcast-1";
const RELAY: &str = "urn:example:wai-bearer:relay-1";
const NOT_HELD: &str = "urn:example:wai-bearer:broadcast-9";
const GROUP_START: &str = "2026-10-21T12:00:00Z";
const GROUP_END: &str = "2026-10-21T12:00:02Z";
const SENT: &str = "2026-10-21T12:00:03Z";
const RECONCILED: &str = "2026-10-22T08:00:00Z";
/// The instant the holdings records state their holdings at.
const T: u64 = 1_793_692_800_000;
/// An hour: when they expire.
const HOUR: u64 = 3_600_000;
fn key(seed: u8) -> SigningKey {
SigningKey::from_bytes(&[seed; 32])
}
fn pk(seed: u8) -> [u8; 32] {
key(seed).verifying_key().to_bytes()
}
/// One WAI envelope with an opaque payload, which no case decodes.
fn envelope(tag: &str, i: usize) -> Vec<u8> {
let payload = format!("compact-return {tag} object {i}").into_bytes();
Wai::new(Manifest::new("binary", "improve", ModelRequirement::new("wai.text.zstd"), Conditioning::new("bytes")), payload)
.to_bytes()
.expect("a small envelope writes")
}
/// A published group: its receipt and its object receipts.
#[derive(Clone)]
struct Publication {
group: GroupReceipt,
objects: Vec<ObjectReceipt>,
}
impl Publication {
fn link(&self) -> [u8; 32] {
self.group.link_hash()
}
/// The bitmap and root of a reconciliation of the objects `idx`.
fn commitment(&self, idx: &[u64]) -> (Vec<u8>, [u8; 32]) {
let leaves: Vec<[u8; 32]> = idx.iter().map(|&i| self.objects[i as usize].leaf()).collect();
(ReconciliationClaim::bitmap(self.group.leaf_count, idx), merkle_root(&leaves))
}
}
fn publish(tag: &str, track: u64, n: usize, labelled: bool) -> Publication {
let mut b = GroupBuilder::default();
for i in 0..n {
let h = crate::jwp_hash::content_hash(&envelope(tag, i));
if labelled {
let class = EnergyProvenance::OnChipCounter {
uncertainty: Uncertainty { relative_ppm: 20_000, window_us: 1_000_000 },
filtering: CounterFiltering::Unknown,
};
b.add(h, 100_000 + 10_000 * i as u64, "edge-1", Some(ObjectLabel::Class(class)));
} else {
b.add(h, 0, "", None);
}
}
let (group, objects) =
b.finish(&key(PUBLISHER), D1, track, 1, GROUP_START, GROUP_END, None, labelled).expect("the corpus's objects compose");
Publication { group, objects }
}
/// The groups: of 4, 3 (labelled leaves), 9 and 20 objects, which the receiver holds, and one of
/// 4 it does not.
struct Pubs {
g4: Publication,
g3: Publication,
g9: Publication,
g20: Publication,
far: Publication,
}
fn pubs() -> Pubs {
Pubs { g4: publish("g4", 1, 4, false), g3: publish("g3", 2, 3, true), g9: publish("g9", 3, 9, false), g20: publish("g20", 4, 20, false), far: publish("far", 5, 4, false) }
}
/// Two `bearer_id`s with one reference: the first pair, in order, of the corpus's twin names.
fn twins() -> (String, String) {
let mut seen: BTreeMap<[u8; 4], u32> = BTreeMap::new();
for i in 0u32.. {
let id = format!("urn:example:wai-bearer:twin-{i}");
if let Some(j) = seen.insert(bearer_ref(&id), i) {
return (format!("urn:example:wai-bearer:twin-{j}"), id);
}
}
unreachable!("a 32-bit reference repeats")
}
// ───────────────────────────── contexts ─────────────────────────────
/// One group of a context as the corpus writes it: the group's name, the positions of the object
/// receipts held under it and its bearer instances.
type GroupLine = (&'static str, Vec<u64>, Vec<String>);
fn context_json(groups: &[GroupLine], signers: &[([u8; 32], &str)]) -> Value {
json!({
"groups": groups.iter().map(|(g, objects, bearers)| json!({ "group": g, "objects": objects, "bearers": bearers })).collect::<Vec<_>>(),
"signers": signers.iter().map(|(k, id)| json!({ "key": hex(k), "signer_id": id })).collect::<Vec<_>>(),
})
}
fn all(n: u64) -> Vec<u64> {
(0..n).collect()
}
/// The receiver's groups, with or without their object receipts and instances.
fn receiver_groups(objects: bool, bearers: bool) -> Vec<GroupLine> {
let o = |n: u64| if objects { all(n) } else { Vec::new() };
let b = |ids: &[&str]| if bearers { ids.iter().map(|s| s.to_string()).collect() } else { Vec::new() };
vec![("g4", o(4), b(&[MCAST, BCAST])), ("g3", o(3), b(&[RELAY])), ("g9", o(9), b(&[BCAST])), ("g20", o(20), b(&[BCAST]))]
}
fn receiver_signers() -> Vec<([u8; 32], &'static str)> {
vec![(pk(SINK_B), SINK_B_ID), (pk(HOLDER), HOLDER_ID)]
}
fn contexts() -> Vec<(&'static str, Value)> {
let (twin_a, twin_b) = twins();
let mut twin_groups = receiver_groups(true, true);
twin_groups[0].2.extend([twin_a.clone(), twin_b]);
let mut alone = receiver_groups(true, true);
alone[0].2.push(twin_a);
let mut twin_signers = receiver_signers();
twin_signers.push((pk(SINK_B), SINK_B_OTHER_ID));
twin_signers.push((pk(HOLDER), HOLDER_OTHER_ID));
// g4 without the broadcast, which g9 and g20 keep.
let mut elsewhere = receiver_groups(true, true);
elsewhere[0].2.retain(|b| b != BCAST);
vec![
("receiver", context_json(&receiver_groups(true, true), &receiver_signers())),
("no_objects", context_json(&receiver_groups(false, true), &receiver_signers())),
("no_bearers", context_json(&receiver_groups(true, false), &receiver_signers())),
("no_signers", context_json(&receiver_groups(true, true), &[])),
("empty", context_json(&[], &[])),
("one_twin", context_json(&alone, &receiver_signers())),
("twin_bearers", context_json(&twin_groups, &receiver_signers())),
("twin_signer", context_json(&receiver_groups(true, true), &twin_signers)),
("bearer_elsewhere", context_json(&elsewhere, &receiver_signers())),
]
}
fn groups_json(p: &Pubs) -> Value {
let mut m = Map::new();
for (name, g) in [("g4", &p.g4), ("g3", &p.g3), ("g9", &p.g9), ("g20", &p.g20), ("far", &p.far)] {
m.insert(name.into(), json!({ "receipt": g.group.to_json(), "objects": g.objects.iter().map(|o| o.to_json()).collect::<Vec<_>>() }));
}
Value::Object(m)
}
/// A context from the corpus's form: each group by name from `groups`, with the object receipts
/// at the positions it lists.
pub fn context_of(v: &Value, groups: &Value) -> Result<Context, String> {
let mut ctx = Context::default();
for g in v["groups"].as_array().ok_or("groups")? {
let name = g["group"].as_str().ok_or("a group's name")?;
let entry = &groups[name];
let receipt = GroupReceipt::from_json(&entry["receipt"]).map_err(|e| format!("{name}: {e:?}"))?;
let held = entry["objects"].as_array().ok_or("objects")?;
let mut objects = Vec::new();
for i in g["objects"].as_array().ok_or("a group's objects")? {
let o = held.get(i.as_u64().ok_or("a position")? as usize).ok_or("a position past the objects")?;
objects.push(ObjectReceipt::from_json(o).map_err(|e| format!("{name}: {e:?}"))?);
}
let bearers = g["bearers"]
.as_array()
.ok_or("bearers")?
.iter()
.map(|b| b.as_str().map(String::from).ok_or_else(|| "a bearer_id".to_string()))
.collect::<Result<Vec<_>, _>>()?;
ctx.add_group(receipt, objects, bearers);
}
for s in v["signers"].as_array().ok_or("signers")? {
let key = s["key"].as_str().and_then(unhex).and_then(|b| <[u8; 32]>::try_from(b).ok()).ok_or("a signer's key")?;
ctx.signers.push(Signer { key, signer_id: s["signer_id"].as_str().ok_or("a signer_id")?.to_string() });
}
Ok(ctx)
}
// ───────────────────────────── records ─────────────────────────────
fn reconciliation_body(p: &Publication, bearer: Bearer, id: &str, idx: &[u64]) -> ReconciliationClaim {
let (received, received_root) = p.commitment(idx);
ReconciliationClaim { bearer, bearer_id: id.into(), leaf_count: p.group.leaf_count, received, received_root, attests: ATTESTS_RECEIVED.into() }
}
/// A reconciliation a verifier accepts at jwp-receipts §3.1 steps 1 to 4.
fn reconcile(seed: u8, id: &str, p: &Publication, body: ReconciliationClaim, seq: u64, at: &str, parent: Option<[u8; 32]>) -> ClaimReceipt {
ClaimReceipt::seal(&key(seed), id, SubjectType::Receipt, p.link(), seq, at, ClaimBody::Reconciliation(body), parent)
.expect("a reconciliation the corpus means to pass steps 1-4")
}
/// One that breaks a rule of steps 1, 2 or 4, validly signed.
fn reconcile_unchecked(seed: u8, subject_type: SubjectType, p: &Publication, body: ReconciliationClaim) -> ClaimReceipt {
ClaimReceipt::seal_unchecked(&key(seed), did_key(&pk(seed)), subject_type, p.link(), 0, RECONCILED, ClaimBody::Reconciliation(body), None)
.expect("a canonical form")
}
/// The records the cases are built from.
struct Records {
/// Sink A, a fresh did:key: every object of g4, over the broadcast.
whole: ClaimReceipt,
/// Sink B, enrolled: objects 0 and 2 of g4, over the multicast.
partial: ClaimReceipt,
/// Sink B's next claim, chained to the last: objects 0 to 7 of g9.
chained: ClaimReceipt,
/// Sink C, under a label: object 1 of g3, whose leaves are labelled, over the relay.
labelled: ClaimReceipt,
/// Sink B's third: every object of g20.
whole_twenty: ClaimReceipt,
/// Sink D, a fresh did:key: every object of g4, over the first twin instance.
twin: ClaimReceipt,
/// The holder, whose key the receiver holds: objects 0 and 8 of g9, g20 whole, object 1 of a
/// group the receiver does not hold.
held: Holdings,
/// A neighbour the receiver does not know: g4 whole, and the group it does not hold whole.
neighbour: Holdings,
}
fn records(p: &Pubs) -> Records {
let (twin_a, _) = twins();
let whole = reconcile(SINK_A, &did_key(&pk(SINK_A)), &p.g4, reconciliation_body(&p.g4, Bearer::Unidirectional, BCAST, &all(4)), 0, RECONCILED, None);
let partial = reconcile(SINK_B, SINK_B_ID, &p.g4, reconciliation_body(&p.g4, Bearer::Multicast, MCAST, &[0, 2]), 0, RECONCILED, None);
let chained = reconcile(SINK_B, SINK_B_ID, &p.g9, reconciliation_body(&p.g9, Bearer::Unidirectional, BCAST, &all(8)), 1, RECONCILED, Some(partial.link_hash()));
let whole_twenty =
reconcile(SINK_B, SINK_B_ID, &p.g20, reconciliation_body(&p.g20, Bearer::Unidirectional, BCAST, &all(20)), 2, RECONCILED, Some(chained.link_hash()));
let labelled = reconcile(SINK_C, SINK_C_ID, &p.g3, reconciliation_body(&p.g3, Bearer::Bidirectional, RELAY, &[1]), 0, RECONCILED, None);
let twin = reconcile(SINK_D, &did_key(&pk(SINK_D)), &p.g4, reconciliation_body(&p.g4, Bearer::Multicast, &twin_a, &all(4)), 0, RECONCILED, None);
let e = |g: &Publication, held: &[u64]| Entry::of(g.link(), g.group.leaf_count, held).expect("a case's entry holds something");
let held = Holdings::seal(&key(HOLDER), T, T + HOUR, vec![e(&p.g9, &[0, 8]), e(&p.g20, &all(20)), e(&p.far, &[1])]);
let neighbour = Holdings::seal(&key(NEIGHBOUR), T, T + HOUR, vec![e(&p.g4, &all(4)), e(&p.far, &all(4))]);
Records { whole, partial, chained, labelled, whole_twenty, twin, held, neighbour }
}
fn rec(c: &ClaimReceipt) -> Record {
Record::Reconciliation(c.to_json())
}
fn hold(h: &Holdings) -> Record {
Record::Holdings(h.to_bytes())
}
fn kind_of(r: &Record) -> &'static str {
match r {
Record::Reconciliation(_) => "reconciliation",
Record::Holdings(_) => "holdings",
}
}
fn record_value(r: &Record) -> Value {
match r {
Record::Reconciliation(s) => Value::String(s.clone()),
Record::Holdings(b) => Value::String(hex(b)),
}
}
fn record_of(kind: &str, v: &Value) -> Result<Record, String> {
let s = v.as_str().ok_or("a record")?;
match kind {
"reconciliation" => Ok(Record::Reconciliation(s.to_string())),
"holdings" => unhex(s).map(Record::Holdings).ok_or_else(|| "not hex".to_string()),
_ => Err(format!("kind {kind}")),
}
}
fn reject(r: Rejection) -> String {
format!("reject step={} reason={}", r.step, r.reason.label())
}
/// What a record's own text says of it: a reconciliation through jwp-receipts §3.1 steps 1 to 4,
/// then step 6 by bearer-receipts §5.3 against the context's group and objects; a holdings record
/// read under holdings §4.4.
pub fn verdict(record: &Record, ctx: &Context) -> String {
match record {
Record::Reconciliation(text) => {
let c = match ClaimReceipt::from_json(text) {
Ok(c) => c,
Err(r) => return reject(r),
};
if let Err(r) = c.verify() {
return reject(r);
}
let Some(g) = ctx.group_by_link(&c.subject) else { return "reject step=6 reason=context_invalid".into() };
let objects: Vec<&ObjectReceipt> = g.objects.iter().collect();
match c.check_reconciliation_against(&g.receipt, &objects) {
Ok(()) => "accept".into(),
Err(r) => reject(r),
}
}
Record::Holdings(b) => match Holdings::read(b) {
Ok(h) => format!("read digest={}", hex(&h.digest())),
Err(r) => format!("reject reason={r}"),
},
}
}
/// The contexts, built.
fn built(contexts: &[(&str, Value)], groups: &Value) -> BTreeMap<String, Context> {
contexts.iter().map(|(n, v)| (n.to_string(), context_of(v, groups).expect("the generator's contexts read"))).collect()
}
fn compact(r: &Record, ctx: &Context) -> Vec<u8> {
encode(r, ctx).expect("a record the corpus means to have a compact form")
}
fn record_cases(r: &Records, ctx: &BTreeMap<String, Context>) -> Vec<Value> {
let case = |name: &str, description: &str, record: Record, verdict: &str| {
json!({
"name": name,
"description": description,
"context": "receiver",
"kind": kind_of(&record),
"record": record_value(&record),
"compact": hex(&compact(&record, &ctx["receiver"])),
"outcome": { "verdict": verdict },
})
};
let digest = |h: &Holdings| format!("read digest={}", hex(&h.digest()));
vec![
case("whole_did_key", "Every object of a group of four over a broadcast, signed under a fresh did:key: signer form 0, no bitmap.", rec(&r.whole), "accept"),
case("partial_enrolled", "Two objects of four over a multicast, by a sink whose key the receiver holds: signer form 2.", rec(&r.partial), "accept"),
case("partial_chained", "The same sink's next claim, its parent carried: eight objects of nine, a bitmap's last byte part used.", rec(&r.chained), "accept"),
case("whole_twenty", "Its third: every object of twenty.", rec(&r.whole_twenty), "accept"),
case("labelled_label_id", "One object of a group of labelled leaves over a relay, by a sink signing under a label: signer form 1.", rec(&r.labelled), "accept"),
case("holdings_by_reference", "A holder the receiver knows: its key, and two of its three groups, by reference; the third inline.", hold(&r.held), &digest(&r.held)),
case("holdings_inline_holder", "A holder the receiver does not know, inline: one group by reference, one inline, both whole.", hold(&r.neighbour), &digest(&r.neighbour)),
]
}
fn expand_outcome(record: &Record, verdict: &str) -> Value {
json!({ "outcome": "expand", "kind": kind_of(record), "record": record_value(record), "verdict": verdict })
}
fn refuse(reason: &str) -> Value {
json!({ "outcome": "refuse", "reason": reason })
}
/// The uint bytes of `v`.
fn uint_bytes(v: u64) -> Vec<u8> {
let mut b = Vec::new();
put_uint(&mut b, v);
b
}
fn expansion_cases(r: &Records, ctx: &BTreeMap<String, Context>) -> Vec<Value> {
let recv = &ctx["receiver"];
let mut out = Vec::new();
let mut case = |name: &str, description: &str, context: &str, bytes: &[u8], outcome: Value| {
out.push(json!({ "name": name, "description": description, "context": context, "compact": hex(bytes), "outcome": outcome }));
};
let whole = compact(&rec(&r.whole), recv);
let partial = compact(&rec(&r.partial), recv);
let labelled = compact(&rec(&r.labelled), recv);
let held = compact(&hold(&r.held), recv);
let neighbour = compact(&hold(&r.neighbour), recv);
let accept = "accept";
let held_digest = format!("read digest={}", hex(&r.held.digest()));
let neighbour_digest = format!("read digest={}", hex(&r.neighbour.digest()));
// Every form reads: a value the receiver holds, written inline, expands to the same record.
case("signer_inline_though_held", "Sink B's claim with its key and signer_id inline (form 1), although the receiver holds them.", "receiver",
&compact(&rec(&r.partial), &ctx["no_signers"]), expand_outcome(&rec(&r.partial), accept));
case("holder_inline_though_held", "The holder's key inline, although the receiver holds it.", "receiver",
&compact(&hold(&r.held), &ctx["no_signers"]), expand_outcome(&hold(&r.held), &held_digest));
case("groups_inline_though_held", "Every group inline, although the receiver holds them.", "receiver",
&compact(&hold(&r.neighbour), &ctx["empty"]), expand_outcome(&hold(&r.neighbour), &neighbour_digest));
case("whole_needs_no_objects", "A whole reconciliation, against a receiver that holds no object receipt: its root is the group's.", "no_objects",
&whole, expand_outcome(&rec(&r.whole), accept));
// A compact record adds no trust: a changed byte expands, and the record refuses itself.
let changed = |b: &[u8], at: usize, x: u8| {
let mut c = b.to_vec();
c[at] ^= x;
c
};
let refused_record = |b: &[u8], reason: &str| {
let rec = expand(b, recv).expect("a changed record that still expands");
expand_outcome(&rec, reason)
};
let sig = changed(&whole, whole.len() - 1, 0x01);
case("signature_changed", "A bit of the signature flipped: it expands, and the claim's signature does not verify.", "receiver", &sig,
refused_record(&sig, "reject step=3 reason=signature_verify_failed"));
let seq = changed(&whole, 14, 0x01);
case("seq_changed", "seq 1 for 0: it expands to a claim its signer did not sign.", "receiver", &seq,
refused_record(&seq, "reject step=3 reason=signature_verify_failed"));
let class = changed(&whole, 1, 0x03);
case("bearer_class_changed", "The bearer class multicast for unidirectional: it expands to a claim its signer did not sign.", "receiver",
&class, refused_record(&class, "reject step=3 reason=signature_verify_failed"));
let hsig = changed(&held, held.len() - 1, 0x01);
case("holdings_signature_changed", "A bit of the holder's signature flipped.", "receiver", &hsig,
refused_record(&hsig, "reject reason=holdings_signature_invalid"));
// §13.3 and §13.4: the reader's rules.
let m = refuse(MALFORMED);
let with = |b: &[u8], at: usize, x: u8| {
let mut c = b.to_vec();
c[at] = x;
c
};
let splice = |b: &[u8], at: usize, len: usize, new: &[u8]| [&b[..at], new, &b[at + len..]].concat();
case("kind_unknown", "kind 0x03.", "receiver", &with(&whole, 0, 0x03), m.clone());
case("form_reserved_bit", "form with 0x40 set.", "receiver", &with(&whole, 1, whole[1] | 0x40), m.clone());
case("bearer_three", "A bearer class of 3.", "receiver", &with(&whole, 1, whole[1] | 0x03), m.clone());
case("signer_form_three", "A signer form of 3.", "receiver", &with(&whole, 1, whole[1] | 0x18), m.clone());
case("truncated", "The last byte of the signature missing.", "receiver", &whole[..whole.len() - 1], m.clone());
case("trailing", "A byte after the signature.", "receiver", &[&whole[..], &[0]].concat(), m.clone());
// seq is at 14: kind, form, group (8), bearer (4).
case("seq_not_minimal", "seq 0 written in two bytes.", "receiver", &splice(&whole, 14, 1, &[0x80, 0x00]), m.clone());
case("seq_past_bound", "seq 2^53.", "receiver", &splice(&whole, 14, 1, &uint_bytes(1 << 53)), m.clone());
case("uint_nine_bytes", "seq in nine bytes.", "receiver", &splice(&whole, 14, 1, &[0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x01]), m.clone());
let at_len = uint_bytes(instant_of(RECONCILED).expect("an instant")).len();
case("issued_at_past_bound", "issued_at one second past 9999-12-31T23:59:59Z.", "receiver",
&splice(&whole, 15, at_len, &uint_bytes(MAX_SECONDS + 1)), m.clone());
{
// Form 1 with the key's own did:key: that is form 0's spelling.
let id = did_key(&pk(SINK_A));
let key_end = 15 + at_len + 32;
let mut b = with(&whole, 1, whole[1] | 0x08);
b = splice(&b, key_end, 0, &[uint_bytes(id.len() as u64), id.into_bytes()].concat());
case("signer_id_is_did_key", "Signer form 1 naming the key's own did:key, which is form 0.", "receiver", &b, m.clone());
}
{
// Sink C's label: kind, form, group, bearer, seq (1), issued_at, key (32), then its length.
let id_at = 15 + at_len + 32 + 1;
case("signer_id_not_utf8", "A signer_id that is not UTF-8.", "receiver", &with(&labelled, id_at, 0xff), m.clone());
}
// The bitmap of objects 0 and 2 of four is the byte before the signature.
let bitmap_at = partial.len() - 65;
case("received_every_bit", "A partial bitmap of every object of four: the whole form's spelling.", "receiver", &with(&partial, bitmap_at, 0xf0), m.clone());
case("received_no_bit", "A partial bitmap of no object.", "receiver", &with(&partial, bitmap_at, 0x00), m.clone());
case("received_past_leaf_count", "Object 4 of a group of four set.", "receiver", &with(&partial, bitmap_at, 0xa8), m.clone());
case("holdings_holder_form", "A holder form of 2.", "receiver", &with(&held, 1, 0x02), m.clone());
let n_at = 2 + 8 + uint_bytes(T).len() + uint_bytes(HOUR).len();
case("holdings_entry_form", "An entry's form with 0x04 set.", "receiver", &with(&held, n_at + 1, held[n_at + 1] | 0x04), m.clone());
case("holdings_count_past_u32", "An entry count of 2^32, which WHD1 cannot state.", "receiver", &splice(&held, n_at, 1, &uint_bytes(1 << 32)), m.clone());
case("holdings_truncated", "The last byte of the holder's signature missing.", "receiver", &held[..held.len() - 1], m.clone());
case("holdings_trailing", "A byte after the holder's signature.", "receiver", &[&held[..], &[0]].concat(), m.clone());
// as_of_ms is at 10: kind, form, holder (8); window_ms follows it.
let window_at = 2 + 8 + uint_bytes(T).len();
case("holdings_as_of_past_bound", "as_of_ms 2^53.", "receiver", &splice(&held, 10, uint_bytes(T).len(), &uint_bytes(1 << 53)), m.clone());
{
let mut longer = uint_bytes(HOUR);
*longer.last_mut().expect("a byte") |= 0x80;
longer.push(0x00);
case("holdings_window_not_minimal", "window_ms written in one byte more than its fewest.", "receiver",
&splice(&held, window_at, uint_bytes(HOUR).len(), &longer), m.clone());
}
{
// The entry of the group the receiver does not hold is inline: its link hash, then its
// leaf_count.
let e = r.held.entries.iter().find(|e| recv.group_by_link(&e.group).is_none()).expect("an entry written inline");
let at = held.windows(32).position(|w| w == &e.group[..]).expect("the inline group") + 32;
case("holdings_leaf_count_nine_bytes", "An inline group's leaf_count in nine bytes.", "receiver",
&splice(&held, at, uint_bytes(e.leaf_count).len(), &[0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x01]), m.clone());
}
// §13.2: references.
let unknown = refuse(REFERENCE_UNKNOWN);
case("group_unknown", "A group the receiver does not hold.", "empty", &whole, unknown.clone());
case("bearer_unknown", "A bearer instance of the group the receiver holds no declaration of.", "no_bearers", &whole, unknown.clone());
case("bearer_of_other_groups", "Sink A's claim over the broadcast, an instance the receiver holds for g9 and g20 but not for g4.",
"bearer_elsewhere", &whole, unknown.clone());
case("signer_unknown", "A signer by reference the receiver does not hold.", "no_signers", &partial, unknown.clone());
case("holder_unknown", "A holder by reference the receiver does not hold.", "no_signers", &held, unknown.clone());
case("holdings_group_unknown", "An entry's group by reference the receiver does not hold.", "empty", &neighbour, unknown);
let twin = compact(&rec(&r.twin), &ctx["one_twin"]);
case("bearer_resolves_alone", "Sink D's claim over the first of two instances with one reference, the receiver holding that one alone.", "one_twin",
&twin, expand_outcome(&rec(&r.twin), accept));
case("bearer_ambiguous", "The same bytes, the receiver holding both instances: refused, not resolved by choosing.", "twin_bearers", &twin,
refuse(REFERENCE_AMBIGUOUS));
case("signer_ambiguous", "Sink B's key held under two signer_ids.", "twin_signer", &partial, refuse(REFERENCE_AMBIGUOUS));
case("holder_under_two_ids", "The holder by reference, its key held under two signer_ids: one key, since a holdings record names no signer_id.",
"twin_signer", &held, expand_outcome(&hold(&r.held), &held_digest));
case("objects_not_held", "A partial reconciliation, against a receiver that holds no object receipt to derive its root from.", "no_objects",
&partial, refuse(OBJECTS_NOT_HELD));
out
}
fn encoding_cases(p: &Pubs, r: &Records) -> Vec<Value> {
let mut out = Vec::new();
let mut case = |name: &str, description: &str, context: &str, record: Record| {
out.push(json!({
"name": name,
"description": description,
"context": context,
"kind": kind_of(&record),
"record": record_value(&record),
"outcome": refuse(NOT_ENCODABLE),
}));
};
let at = |t: &str| {
rec(&reconcile(SINK_A, &did_key(&pk(SINK_A)), &p.g4, reconciliation_body(&p.g4, Bearer::Unidirectional, BCAST, &all(4)), 0, t, None))
};
case("issued_at_fraction", "issued_at with a fraction of a second.", "receiver", at("2026-10-22T08:00:00.5Z"));
case("issued_at_offset", "issued_at with an offset.", "receiver", at("2026-10-22T08:00:00+00:00"));
case("issued_at_lower_case", "issued_at with t and z in lower case.", "receiver", at("2026-10-22t08:00:00z"));
case("issued_at_leap_second", "issued_at at a second of 60.", "receiver", at("2026-10-22T08:00:60Z"));
case("issued_at_before_1970", "issued_at before 1970.", "receiver", at("1969-12-31T23:59:59Z"));
case("not_canonical_text", "The claim with a space after its first brace: not its canonical form.", "receiver",
Record::Reconciliation(r.whole.to_json().replacen('{', "{ ", 1)));
let transmission = ClaimReceipt::seal(
&key(PUBLISHER),
D1,
SubjectType::Receipt,
p.g4.link(),
0,
SENT,
ClaimBody::Transmission(TransmissionClaim { bearer: Bearer::Unidirectional, bearer_id: BCAST.into(), energy: None, attests: "transmitted".into() }),
None,
)
.expect("a transmission claim");
case("transmission_claim", "A transmission claim: this profile writes reconciliations only.", "receiver", rec(&transmission));
case("group_not_held", "A reconciliation of a group the receiver does not hold.", "receiver",
rec(&reconcile(SINK_A, &did_key(&pk(SINK_A)), &p.far, reconciliation_body(&p.far, Bearer::Unidirectional, BCAST, &all(4)), 0, RECONCILED, None)));
case("bearer_not_held", "A reconciliation over an instance of the group the receiver holds no declaration of.", "receiver",
rec(&reconcile(SINK_A, &did_key(&pk(SINK_A)), &p.g4, reconciliation_body(&p.g4, Bearer::Unidirectional, NOT_HELD, &all(4)), 0, RECONCILED, None)));
let mut five = reconciliation_body(&p.g4, Bearer::Unidirectional, BCAST, &[0, 1, 2, 3]);
five.leaf_count = 5;
five.received = vec![0xf8];
case("leaf_count_mismatch", "A leaf count of 5 for a group of four.", "receiver", rec(&reconcile_unchecked(SINK_A, SubjectType::Receipt, &p.g4, five)));
let largest = r.whole.to_json().replacen("\"leaf_count\":4,", "\"leaf_count\":9007199254740991,", 1);
assert_ne!(largest, r.whole.to_json(), "the claim states its leaf_count");
case("leaf_count_largest", "Sink A's claim with a leaf_count of 2^53 − 1 beside its bitmap of one byte: not the group's, and nothing is sized by it.",
"receiver", Record::Reconciliation(largest));
let mut root = reconciliation_body(&p.g4, Bearer::Multicast, MCAST, &[0, 2]);
root.received_root = [0; 32];
case("root_mismatch", "A partial reconciliation whose root is not the one its objects give.", "receiver",
rec(&reconcile_unchecked(SINK_A, SubjectType::Receipt, &p.g4, root)));
let mut rendered = reconciliation_body(&p.g4, Bearer::Unidirectional, BCAST, &all(4));
rendered.attests = "rendered".into();
case("attests_other", "attests \"rendered\".", "receiver", rec(&reconcile_unchecked(SINK_A, SubjectType::Receipt, &p.g4, rendered)));
let mut none = reconciliation_body(&p.g4, Bearer::Unidirectional, BCAST, &[]);
none.received_root = merkle_root(&[]);
case("no_bit_set", "A reconciliation of no object.", "receiver", rec(&reconcile_unchecked(SINK_A, SubjectType::Receipt, &p.g4, none)));
case("subject_type_object", "subject_type \"object\", which the class does not admit.", "receiver",
rec(&reconcile_unchecked(SINK_A, SubjectType::Object, &p.g4, reconciliation_body(&p.g4, Bearer::Unidirectional, BCAST, &all(4)))));
case("objects_not_held_by_sender", "A partial reconciliation, the sender's context holding no object receipt to derive its root from.", "no_objects",
rec(&r.partial));
let mut bad = r.held.to_bytes();
let n = bad.len() - 1;
bad[n] ^= 0x01;
case("holdings_unreadable", "A holdings record whose signature does not verify.", "receiver", Record::Holdings(bad));
out
}
fn message_cases(r: &Records, recv: &Context) -> Vec<Value> {
let whole = compact(&rec(&r.whole), recv);
let held = compact(&hold(&r.held), recv);
let chained = compact(&rec(&r.chained), recv);
let case = |name: &str, description: &str, record: &[u8], number: u8, frame: usize| {
let fragments = cut(record, number, frame).expect("a frame the corpus cuts for");
json!({
"name": name,
"description": description,
"record": hex(record),
"message": number,
"frame": frame,
"outcome": { "fragments": fragments.iter().map(|f| hex(f)).collect::<Vec<_>>() },
})
};
vec![
case("whole_frame_11", "whole_did_key's compact record in frames of 11 bytes.", &whole, 0x01, 11),
case("whole_frame_13", "The same in frames of 13 bytes.", &whole, 0x02, 13),
case("whole_frame_51", "The same in frames of 51 bytes.", &whole, 0x03, 51),
case("whole_frame_140", "The same in one frame of 140 bytes.", &whole, 0x04, 140),
case("holdings_frame_51", "holdings_by_reference's in frames of 51 bytes.", &held, 0x05, 51),
case("holdings_frame_222", "The same in one frame of 222 bytes.", &held, 0x06, 222),
case("chained_frame_222", "partial_chained's in one frame of 222 bytes.", &chained, 0x07, 222),
]
}
fn reassembly_cases(r: &Records, recv: &Context) -> Vec<Value> {
let whole = compact(&rec(&r.whole), recv);
let held = compact(&hold(&r.held), recv);
let chained = compact(&rec(&r.chained), recv);
// Sink D's compact record has the same length as sink A's, so cut under one number in one
// frame its fragments carry the same headers.
let other = {
let mut ctx = recv.clone();
ctx.groups[0].bearers.push(twins().0);
compact(&rec(&r.twin), &ctx)
};
assert_eq!(other.len(), whole.len(), "two records of one length");
let f11 = cut(&whole, 0x01, 11).expect("frames of 11");
let f13 = cut(&whole, 0x02, 13).expect("frames of 13");
let h51 = cut(&held, 0x05, 51).expect("frames of 51");
let one = cut(&chained, 0x07, 222).expect("one frame");
let same_header = cut(&other, 0x01, 11).expect("frames of 11");
let other_number = cut(&other, 0x09, 11).expect("frames of 11");
let other_count = cut(&held, 0x01, 11).expect("frames of 11");
assert_ne!(other_count[0][2], f11[0][2], "another count");
let mut out = Vec::new();
let mut case = |name: &str, description: &str, fragments: Vec<Vec<u8>>, outcome: Value| {
out.push(json!({ "name": name, "description": description, "fragments": fragments.iter().map(|f| hex(f)).collect::<Vec<_>>(), "outcome": outcome }));
};
let ok = |record: &[u8]| json!({ "outcome": "reassembled", "record": hex(record) });
case("in_order", "whole_frame_11's fragments in order.", f11.clone(), ok(&whole));
let mut reversed = f13.clone();
reversed.reverse();
case("arrival_reversed", "whole_frame_13's fragments last first: each is placed by its index.", reversed, ok(&whole));
let mut shuffled = h51.clone();
shuffled.rotate_left(2);
shuffled.swap(0, 1);
case("arrival_shuffled", "holdings_frame_51's fragments in another order.", shuffled, ok(&held));
case("one_fragment", "chained_frame_222's one fragment.", one, ok(&chained));
let without = |i: usize| {
let mut f = f11.clone();
f.remove(i);
f
};
case("missing_middle", "Fragment 4 of whole_frame_11 missing.", without(4), refuse(FRAGMENT_MISSING));
case("missing_last", "Its last fragment missing.", without(f11.len() - 1), refuse(FRAGMENT_MISSING));
case("none", "No fragment.", Vec::new(), refuse(FRAGMENT_MISSING));
let mut twice = f11.clone();
twice.insert(3, f11[2].clone());
case("delivered_twice", "Fragment 2 twice, byte for byte: one fragment, held once.", twice, ok(&whole));
let mut differing = f11.clone();
let mut copy = f11[2].clone();
copy[HEADER] ^= 0x01;
differing.push(copy);
case("repeated_other_bytes", "A second fragment 2, with other data.", differing, refuse(FRAGMENT_REPEATED));
let mut relabelled = f11.clone();
relabelled[3][1] = 4;
case("relabelled_onto_another", "Fragment 3 relabelled as fragment 4.", relabelled, refuse(FRAGMENT_REPEATED));
let mut exchanged = f11.clone();
let (a, b) = (f11[3][HEADER..].to_vec(), f11[4][HEADER..].to_vec());
exchanged[3] = [&f11[3][..HEADER], &b[..]].concat();
exchanged[4] = [&f11[4][..HEADER], &a[..]].concat();
case("reordered_data_exchanged", "The data of fragments 3 and 4 exchanged, their headers kept: a reordered fragment.", exchanged,
refuse(FRAGMENT_CHECK_FAILED));
let mut swapped_index = f11.clone();
swapped_index[3][1] = 4;
swapped_index[4][1] = 3;
case("reordered_index_exchanged", "The indices of fragments 3 and 4 exchanged: each placed where the other was cut from.", swapped_index,
refuse(FRAGMENT_CHECK_FAILED));
let mut foreign_number = f11.clone();
foreign_number[2] = other_number[2].clone();
case("foreign_number", "Fragment 2 of another message, under another number.", foreign_number, refuse(FRAGMENT_FOREIGN));
let mut foreign_count = f11.clone();
foreign_count[2] = other_count[2].clone();
case("foreign_count", "Fragment 2 of another message under the same number, with another count.", foreign_count, refuse(FRAGMENT_FOREIGN));
let mut foreign_same = f11.clone();
foreign_same[2] = same_header[2].clone();
case("foreign_same_header", "Fragment 2 of another message under the same number and count: only the check finds it.", foreign_same,
refuse(FRAGMENT_CHECK_FAILED));
let mut flipped = f11.clone();
flipped[5][HEADER + 2] ^= 0x10;
case("data_changed", "A bit of fragment 5's data flipped.", flipped, refuse(FRAGMENT_CHECK_FAILED));
let mut header_only = f11.clone();
header_only[1].truncate(HEADER);
case("header_only", "A fragment of its header alone.", header_only, refuse(FRAGMENT_MALFORMED));
let mut count_zero = f11.clone();
count_zero[0][2] = 0;
case("count_zero", "A fragment with a count of 0.", count_zero, refuse(FRAGMENT_MALFORMED));
let mut past = f11.clone();
let n = past.len() as u8;
past[1][1] = n;
case("index_past_count", "A fragment whose index is its count.", past, refuse(FRAGMENT_MALFORMED));
case("check_without_record", "One fragment holding a check and no record.", vec![[&[0x0a, 0, 1][..], &check(&[])[..]].concat()],
refuse(FRAGMENT_CHECK_FAILED));
out
}
/// The corpus: `vectors.json`, from fixed seeds.
pub fn generate() -> Vec<(String, String)> {
let p = pubs();
let r = records(&p);
let groups = groups_json(&p);
let contexts = contexts();
let ctx = built(&contexts, &groups);
let mut keys = Map::new();
for (name, seed) in [("publisher", PUBLISHER), ("sink_a", SINK_A), ("sink_b", SINK_B), ("sink_c", SINK_C), ("sink_d", SINK_D), ("holder", HOLDER), ("neighbour", NEIGHBOUR)] {
keys.insert(name.into(), hex(&pk(seed)).into());
}
let doc = json!({
"about": "Conformance corpus of extensions/bearer-receipts.md §13, the compact return profile: reconciliations (§5) and \
holdings records (holdings §4) in compact form against a receiver's context (§13.1 to §13.5), and their \
messages in fragments (§13.6, §13.7). Every key is Ed25519 from a fixed 32-byte seed and attests nothing; \
envelopes carry opaque test payloads no case decodes. Each case states the outcome it was written to have, \
declared in the generator. Regenerate with `wai_compact_return_vectors gen`.",
"keys": keys,
"groups": groups,
"contexts": contexts.iter().map(|(n, v)| (n.to_string(), v.clone())).collect::<Map<String, Value>>(),
"records": record_cases(&r, &ctx),
"expansions": expansion_cases(&r, &ctx),
"encodings": encoding_cases(&p, &r),
"messages": message_cases(&r, &ctx["receiver"]),
"reassemblies": reassembly_cases(&r, &ctx["receiver"]),
});
let mut s = serde_json::to_string_pretty(&doc).expect("serialises");
s.push('\n');
vec![("vectors.json".into(), s)]
}
// ───────────────────────────── reading it back ─────────────────────────────
fn bytes_of(v: &Value) -> Result<Vec<u8>, String> {
v.as_str().and_then(unhex).ok_or_else(|| format!("not hex: {v}"))
}
fn strings(v: &Value, out: &mut Vec<String>) {
match v {
Value::String(s) => out.push(s.clone()),
Value::Array(a) => a.iter().for_each(|x| strings(x, out)),
Value::Object(m) => m.values().for_each(|x| strings(x, out)),
_ => {}
}
}
/// Check a corpus's files against this module: each case's outcome, every round trip both ways,
/// every cut reassembling to its record, and every reason of §13.9. Returns the failures.
pub fn verify(files: &BTreeMap<String, String>) -> Vec<String> {
let mut fails = Vec::new();
let Some(text) = files.get("vectors.json") else { return vec!["no vectors.json".into()] };
let doc: Value = match serde_json::from_str(text) {
Ok(d) => d,
Err(e) => return vec![format!("vectors.json: {e}")],
};
let mut contexts: BTreeMap<String, Context> = BTreeMap::new();
for (name, v) in doc["contexts"].as_object().into_iter().flatten() {
match context_of(v, &doc["groups"]) {
Ok(c) => {
contexts.insert(name.clone(), c);
}
Err(e) => fails.push(format!("context {name}: {e}")),
}
}
let ctx = |c: &Value| contexts.get(c["context"].as_str().unwrap_or("")).ok_or_else(|| format!("no context {}", c["context"]));
let empty = Vec::new();
let mut stated = Vec::new();
let mut check = |kind: &str, run: &dyn Fn(&Value) -> Result<Value, String>| {
let cases = doc[kind].as_array().unwrap_or(&empty);
if cases.is_empty() {
fails.push(format!("no {kind} cases"));
}
for c in cases {
let name = c["name"].as_str().unwrap_or("?");
match run(c) {
Err(e) => fails.push(format!("{name}: {e}")),
Ok(got) if got != c["outcome"] => fails.push(format!("{name}: stated {} got {got}", c["outcome"])),
Ok(_) => {}
}
strings(&c["outcome"], &mut stated);
}
};
check("records", &|c| {
let ctx = ctx(c)?;
let record = record_of(c["kind"].as_str().unwrap_or(""), &c["record"])?;
let stated = bytes_of(&c["compact"])?;
let written = encode(&record, ctx).map_err(|r| format!("not written: {r}"))?;
if written != stated {
return Err(format!("written as {}", hex(&written)));
}
let back = expand(&stated, ctx).map_err(|r| format!("not expanded: {r}"))?;
if back != record {
return Err("expands to another record".into());
}
Ok(json!({ "verdict": verdict(&record, ctx) }))
});
check("expansions", &|c| {
let ctx = ctx(c)?;
Ok(match expand(&bytes_of(&c["compact"])?, ctx) {
Ok(record) => expand_outcome(&record, &verdict(&record, ctx)),
Err(r) => refuse(r),
})
});
check("encodings", &|c| {
let record = record_of(c["kind"].as_str().unwrap_or(""), &c["record"])?;
Ok(match encode(&record, ctx(c)?) {
Ok(b) => json!({ "outcome": "encode", "compact": hex(&b) }),
Err(r) => refuse(r),
})
});
check("messages", &|c| {
let record = bytes_of(&c["record"])?;
let number = c["message"].as_u64().and_then(|n| u8::try_from(n).ok()).ok_or("message")?;
let frame = c["frame"].as_u64().ok_or("frame")? as usize;
let fragments = cut(&record, number, frame).ok_or("not cut")?;
if reassemble(&fragments) != Ok(record) {
return Err("its fragments do not reassemble to it".into());
}
Ok(json!({ "fragments": fragments.iter().map(|f| hex(f)).collect::<Vec<_>>() }))
});
check("reassemblies", &|c| {
let fragments: Vec<Vec<u8>> = c["fragments"].as_array().ok_or("fragments")?.iter().map(bytes_of).collect::<Result<_, _>>()?;
Ok(match reassemble(&fragments) {
Ok(record) => json!({ "outcome": "reassembled", "record": hex(&record) }),
Err(r) => refuse(r),
})
});
for code in CODES {
if !stated.iter().any(|s| s == code) {
fails.push(format!("no case gives {code}"));
}
}
fails
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_corpus_reaches_every_stated_outcome() {
let files: BTreeMap<String, String> = generate().into_iter().collect();
let fails = verify(&files);
assert!(fails.is_empty(), "{}", fails.join("\n"));
}
#[test]
fn a_changed_outcome_fails_verification() {
let mut files: BTreeMap<String, String> = generate().into_iter().collect();
let text = files["vectors.json"].replacen("\"fragment_repeated\"", "\"fragment_missing\"", 1);
files.insert("vectors.json".into(), text);
assert!(!verify(&files).is_empty());
}
#[test]
fn the_twins_share_a_reference() {
let (a, b) = twins();
assert_ne!(a, b);
assert_eq!(bearer_ref(&a), bearer_ref(&b));
}
}

View file

@ -850,6 +850,13 @@ pub mod conform_result;
/// cbindgen:ignore
#[cfg(feature = "bearer_receipts")]
pub mod bearer_receipts;
// The compact return profile (extensions/bearer-receipts.md §13): a reconciliation and a holdings
// record in a binary form that names by reference what the receiver holds, expanded to the
// record's own bytes before it is verified, and carried in fragments with a check over the whole
// message. With both records' features, and no feature of its own. Rust-only.
/// cbindgen:ignore
#[cfg(all(feature = "bearer_receipts", feature = "holdings"))]
pub mod compact_return;
// Carousel schedule (extensions/carousel.md): `wai.bearer.carousel` — when each object of a group
// passes on a bearer instance with no return path and with which forward error correction; when a
// receiver within a loss bound holds it, and the passes a receiver listens to. Rust-only.

View file

@ -0,0 +1,99 @@
//! The compact return profile (extensions/bearer-receipts.md §13): the committed corpus
//! `live-conformance/compact-return/` regenerates byte for byte from its fixed seeds, holds no other
//! file, every case in it gets the outcome it states, §13.9 lists exactly the reasons the reference
//! reports, and §13.8's floors are the lengths of the reports of telemetry-aggregation's corpus,
//! read with the reference's DAP reader.
#![cfg(all(feature = "bearer_receipts", feature = "holdings", feature = "telemetry_aggregate"))]
use std::collections::BTreeMap;
use std::path::PathBuf;
use wai::compact_return::{corpus, CODES};
use wai::telemetry_aggregate::{dap, measure::MEASURES};
fn wai_dir() -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("..")
}
fn committed() -> BTreeMap<String, String> {
let dir = wai_dir().join("live-conformance/compact-return");
std::fs::read_dir(&dir)
.expect("the committed corpus")
.flatten()
.map(|e| (e.file_name().to_string_lossy().into_owned(), e.path()))
.filter(|(n, _)| n != "README.md")
.map(|(n, p)| (n, std::fs::read_to_string(p).expect("a text file")))
.collect()
}
fn unhex(s: &str) -> Vec<u8> {
(0..s.len()).step_by(2).map(|i| u8::from_str_radix(&s[i..i + 2], 16).expect("hex")).collect()
}
/// The text of one `###` section of bearer-receipts, from its heading to the next heading.
fn section(heading: &str) -> String {
let text = std::fs::read_to_string(wai_dir().join("extensions/bearer-receipts.md")).expect("the extension");
let at = text.find(&format!("\n### {heading}")).unwrap_or_else(|| panic!("a section {heading}"));
let rest = &text[at + 1..];
let end = rest.find("\n#").map_or(rest.len(), |e| e + 1);
rest[..end].to_string()
}
#[test]
fn the_committed_corpus_regenerates_from_its_seeds() {
let files = committed();
let generated: BTreeMap<String, String> = corpus::generate().into_iter().collect();
let extra: Vec<&String> = files.keys().filter(|k| !generated.contains_key(*k)).collect();
let missing: Vec<&String> = generated.keys().filter(|k| !files.contains_key(*k)).collect();
let differ: Vec<&String> = generated.iter().filter(|(k, v)| files.get(*k).is_some_and(|f| f != *v)).map(|(k, _)| k).collect();
assert!(
extra.is_empty() && missing.is_empty() && differ.is_empty(),
"extra {extra:?}, missing {missing:?}, differ {differ:?}: run `wai_compact_return_vectors gen`"
);
}
#[test]
fn every_case_gets_its_stated_outcome() {
let fails = corpus::verify(&committed());
assert!(fails.is_empty(), "{}", fails.join("\n"));
}
/// §13.9 lists exactly the codes `CODES` has, each once, in order.
#[test]
fn the_extension_lists_the_reasons() {
let listed: Vec<String> =
section("13.9 Reasons").lines().filter(|l| l.starts_with("| `")).filter_map(|l| l.split('`').nth(1).map(String::from)).collect();
let want: Vec<String> = CODES.iter().map(|c| c.to_string()).collect();
assert_eq!(listed, want, "§13.9's table and CODES differ");
}
/// §13.8's floor of each of telemetry-aggregation's measures 1 to 4 is the length of its longest
/// report in that extension's corpus, and every report of one task has that length.
#[test]
fn the_floors_are_the_reports_lengths() {
let rows: Vec<(u8, String, usize)> = section("13.8 ")
.lines()
.filter(|l| l.starts_with("| ") && l.contains('`'))
.map(|l| {
let cells: Vec<&str> = l.trim_matches('|').split('|').map(str::trim).collect();
let code: u8 = cells[0].split(' ').next().and_then(|c| c.parse().ok()).expect("a measure's code");
let name = cells[0].split('`').nth(1).expect("a measure's name").to_string();
let floor: usize = cells[1].replace(',', "").parse().expect("a floor");
(code, name, floor)
})
.collect();
let codes: Vec<u8> = rows.iter().map(|r| r.0).collect();
assert_eq!(codes, vec![1, 2, 3, 4], "§13.8 states the floors of measures 1 to 4");
let text = std::fs::read_to_string(wai_dir().join("live-conformance/aggregate/vectors.json")).expect("the aggregate corpus");
let doc: serde_json::Value = serde_json::from_str(&text).expect("JSON");
for (code, name, floor) in rows {
let named = MEASURES.iter().find(|(c, _)| *c == code).map(|(_, n)| *n);
assert_eq!(named, Some(name.as_str()), "measure {code}'s name");
let flow = doc["flows"].as_array().expect("flows").iter().find(|f| f["name"] == name.as_str()).expect("a flow of the measure");
let upload = unhex(flow["upload"].as_str().expect("an upload"));
let reports = dap::decode_upload(&upload).expect("an upload the reference reads");
assert!(!reports.is_empty());
let lengths: std::collections::BTreeSet<usize> = reports.iter().map(|r| r.encode().len()).collect();
assert_eq!(lengths.len(), 1, "every report of {name} has one length");
assert_eq!(lengths.into_iter().next(), Some(floor), "{name}'s floor");
}
}

View file

@ -15,6 +15,7 @@ import json
import os
import re
import socket
import socketserver
import sys
import threading
import time
@ -55,6 +56,14 @@ class Room(http.server.BaseHTTPRequestHandler):
pass
class Server(http.server.ThreadingHTTPServer):
"""HTTPServer looks up a name for its address before it listens, and nothing here reads the
name. On some hosts that lookup takes 35 s, past what `wai dev` waits for a service."""
def server_bind(self):
socketserver.TCPServer.server_bind(self)
def main(argv):
if argv[:1] in (["keygen"], ["relay-lines"]):
open(argv[1], "a").close()
@ -73,7 +82,7 @@ def main(argv):
while True:
c, _ = s.accept()
c.close()
http.server.ThreadingHTTPServer(listen_addr(config, "listen"), Room).serve_forever()
Server(listen_addr(config, "listen"), Room).serve_forever()
return 0