open-standards/wai/extensions/peer-delivery.md

# WAI Extension: Peer Delivery

> Status: Draft. Registers no capability and no record. It is the exchange by which a sink
> fetches objects from the neighbours that hold them before it spends the backhaul, over the
> records other texts define: a neighbour's holdings record says where to ask
> ([holdings](holdings.md) §5.4), the publisher's receipts say what the answer must be
> ([jwp-receipts](jwp-receipts.md) §3), and the asker's settlement acknowledgement says what was
> served ([settlement](settlement.md) §3). A sink joins by its own choice: asking a peer and
> serving one are each off unless the operator's policy and the viewer's setting both allow it
> (§3). The exchange is eight frames over a link that keeps message boundaries, a WebRTC data
> channel among them, and three requests over HTTP for a neighbour reached that way (§4). An
> asker asks only for an object whose object receipt it already holds, takes nothing a peer
> sent before the bytes verify under that receipt, sets aside a holder that served other bytes
> and a record that stated what its holder did not hold, and goes to the origin for what is
> left (§5). It signs an acknowledgement for what it verified, by holder and by window, so what
> each peer served recomputes from signed records (§6). §9 says what a peer learns of a viewer.
> WAI defines no transport. The reference implementation is the `peer_delivery` module of
> `wai-rs` (feature `peer_delivery`) and, in a page, the `peer` bundle of `wai-web`; the corpus
> is [`live-conformance/peer-delivery/`](../live-conformance/peer-delivery/), and its second
> implementation is `live-conformance/peer_delivery_verify.py`.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.
>
> Texts read for this revision: the WAI texts this one cites; RFC 8831 (§6.1, §6.6: a data
> channel's reliability and its message size) and RFC 8832 (§5.1: the channel's type, label and
> protocol); RFC 8828 (§3, §5.1, §5.2: which addresses a WebRTC endpoint gathers, and its four
> modes); RFC 8826 §4.2.4; `draft-ietf-mmusic-mdns-ice-candidates-03` §1 and §3.1.1; RFC 7574
> §3.2 and §5; RFC 9110 §15.5.5, §15.5.21 and §15.6.4; the W3C *WebRTC: Real-Time Communication
> in Browsers* Recommendation (a data channel's label, protocol, order and largest message, and
> the transport policy `relay` of a connection's configuration); the W3C *Service Workers*
> text, §5.4 (*Cache*); and the Web Incubator Community Group's draft *Local Network Access*
> (§1, §2.1 to §2.3). A clause number of one of them in this text is that text's.

## 1. Scope

In scope: who may ask and who may serve (§3); the exchange between an asker and a holder, over a
link that keeps message boundaries and over HTTP (§4); what an asker does with its neighbours'
records and with each answer (§5); the acknowledgement of what a peer served, its object class
and what a holder checks of it (§6); what a page uses of its browser (§7); and what each party
learns (§9).

Out of scope: how two sinks are introduced to each other and how a link between them is set up,
which is the deployment's; how holdings records reach an asker before it has a link (holdings
§1); how a node chooses what to keep; a price; and a record of service beside settlement's
acknowledgement, which this text does not add.

Relationship to the core spec: this text defines no capability, no payload format and no record
([SPEC](../SPEC.md) §5). Every signed byte it relies on is another text's.

## 2. Terms

- **Asker**, **holder**, **group**, **link hash**, **position**: as [holdings](holdings.md) §2.
  A **peer** is a holder that is a sink too; what this text says of a holder holds for a
  neighbourhood cache as it does for a peer.
- **Origin**: where the asker fetches an object when it asks no holder for it, or none serves
  it: the service it takes delivery from.
- **Link**: a path between an asker and one holder that carries messages whole and in order,
  each way, with none lost. An **HTTP link** is a base URL at which a holder answers the
  requests of §4.5.
- **Session**: one asker's exchanges from when it starts until it stops. What an asker sets
  aside (§5.4) it sets aside for its session.
- **Instant**: milliseconds, as holdings §2. A **window** is settlement's, in microseconds
  ([settlement](settlement.md) §2).
- Integers in frames are unsigned and big-endian, as a holdings record's are.

## 3. Joining

### 3.1 The operator's policy

The operator of the service a sink takes delivery from states, in the configuration it gives
its sinks, four values:

| value | meaning | where the operator states nothing |
|---|---|---|
| `ask` | its sinks may fetch from peers | no |
| `serve` | its sinks may serve peers from what they hold | no |
| `max_object_bytes` | the most bytes of one object a sink takes from a peer | 4,194,304 |
| `relay_only` | its sinks reach peers only through a relay (§9) | no |

How the configuration reaches a sink is the deployment's. A sink with no configuration has an
operator that states nothing.

### 3.2 The viewer's setting

A sink keeps two settings of its own, `ask` and `serve`, each **off** until the person using the
sink turns it on, and each theirs to turn off again. A sink **MUST NOT** turn either on by
itself, and **MUST** show, where it offers them, what §9 says a peer learns.

### 3.3 The rule

A sink **asks** only while the policy's `ask` and its own `ask` are both on, and **serves** only
while the policy's `serve` and its own `serve` are both on. Asking is not serving: either may be
on without the other.

- A sink that may not ask opens no link to a holder, sends no frame and no request of §4.5,
  takes no object from a peer and signs no acknowledgement to one. Every object it wants is the
  origin's (`peer_not_permitted`).
- A sink that may not serve signs no holdings record for its neighbours, accepts no link, and
  answers a `want` it is sent, on a link it already has, with `miss` and code 3.
- A setting turned off takes effect at the next object: an exchange under way may finish or be
  dropped, and nothing new starts.

## 4. The exchange

### 4.1 The link and the version

The exchange is version 1. An asker opens it with `record-want`, which states the version; a
holder that does not speak that version closes the link. Each frame is one message of the link.
On a link, an asker has at most one request outstanding: it sends `record-want`, `want` or `ack`
and waits for the whole reply before it sends another.

### 4.2 Frames

A frame is one byte of type, then its body:

| type | name | from | body |
|---|---|---|---|
| 1 | `record-want` | asker | 1 byte: the version, 1 |
| 2 | `record-head` | holder | `len`, 8 bytes: the bytes of the holder's record, 1 to 1,048,576, which follow in `data` frames |
| 3 | `want` | asker | `group`, 32 bytes, and `index`, 8 bytes: object `index` of the group with that link hash |
| 4 | `head` | holder | `group`, `index`, and `len`, 8 bytes, at least 1: the object's bytes, which follow in `data` frames |
| 5 | `data` | holder | 1 to 16,384 bytes |
| 6 | `miss` | holder | `group`, `index`, and `code`, 1 byte: 1 the holder does not hold the object now, 2 its record does not list it, 3 it will not serve now |
| 7 | `ack` | asker | `record_len`, 4 bytes, at least 1; an acknowledgement of that many bytes (§6.1); then its item list, 1 to 256 items of 48 bytes |
| 8 | `ack-result` | holder | `digest`, 32 bytes, the acknowledgement's (settlement §3.5), and 1 byte: 0 taken, 1 not taken |

A reader reads exactly the bytes of one message as one frame and refuses it
`peer_frame_malformed` when it is empty, its type is not 1 to 8, its body is not of its type's
length, or a field is outside the range the table gives. A `record-want` of another version is
`peer_frame_malformed` in this revision. An object's bytes travel in frames of at most 16,384
bytes because RFC 8831 §6.6 asks a sender that cannot interleave messages to keep each within
16 KB.

### 4.3 The holder

A holder answers `record-want` with `record-head` and the bytes of the one record of its own
that stands (holdings §5.3), and `want` with the first of these that holds:

1. `miss`, code 3, when it may not serve (§3.3) or will not serve now;
2. `miss`, code 2, when that record does not hold the object (holdings §5.4): a holder serves
   over this exchange only what it has told its neighbours it holds;
3. `miss`, code 1, when it does not hold the object's bytes now;
4. `head` with the object's length, then its bytes.

It answers `ack` as §6.3 says. A holder **MUST NOT** send bytes it has not itself verified under
the object's receipt, and learns nothing from doing so: its asker verifies them again (§5.3).
A frame a holder is sent that is not `record-want`, `want` or `ack`, or that arrives before its
reply to the one before is whole, is `peer_frame_unexpected`, and it closes the link.

### 4.4 A reply

The bytes a `record-head` or a `head` announces follow in `data` frames, each of 16,384 bytes
but the last, which holds what remains. An asker holds a reply to what it asked:

- a `head` whose `group` or `index` is not what it asked for, a `data` frame that is not the
  next 16,384 bytes or the last bytes where fewer remain, a frame after the reply is whole, and
  any frame of another type than its request admits: `peer_frame_unexpected`;
- a `head` whose `len` is above `max_object_bytes`: `peer_object_too_large`. The asker takes
  none of its bytes;
- a link that closes, or gives no frame within the time the asker allows, before the reply is
  whole: `peer_link_failed`.

After any of the three the asker closes the link and asks that holder nothing more over it. A
reply is whole when its bytes are exactly `len`; an asker never holds more of one than its head
stated, and it stores nothing of a reply that is not whole.

### 4.5 Over HTTP

A holder reached by HTTP answers, under its base URL:

| request | answer |
|---|---|
| `GET holdings` | 200 with the record's bytes |
| `GET object/<group>/<index>` | 200 with the object's bytes, whose length the response states; 404 for `miss` code 1 or 2; 503 for code 3 |
| `POST ack`, its body an `ack` frame's body | 200 with the acknowledgement's digest when it is taken; 422 when it is not |

`<group>` is the link hash as 64 lowercase hexadecimal digits and `<index>` the position in
decimal with no leading zero. The rules of §4.3 and §4.4 hold with a response for a reply: a
stated length above `max_object_bytes`, or a body that reaches it with no length stated, is
`peer_object_too_large`, and the asker reads no further; any other status, a body shorter or
longer than its stated length, and a request that fails are `peer_link_failed`. A 404 is read
as code 1. An asker's requests carry no credentials, no cookie and no referrer (§7).

## 5. The asker

### 5.1 Its records

An asker holds its neighbours' records in the order it prefers them (holdings §5.4) and judges
them at an instant as holdings §5.1 to §5.3 do, against the group receipts it holds: a record
that is refused, superseded or set aside there takes no part. Of those that stand, it sets
aside the ones §5.4 names. Its own record, where it has one, is in no route: an asker does not
ask itself.

### 5.2 The route

For a wanted object, the asker's **route** is, at an instant:

1. the origin alone, `peer_not_permitted`, when it may not ask (§3.3);
2. the origin alone, `peer_receipt_not_held`, when it does not hold the object's object receipt
   and the group receipt it is included under, both verified (jwp-receipts §3 steps 1, 2 and
   5): it could hold nothing a peer sent for it to a signed digest, so it asks no peer for it;
3. otherwise the holders whose record stands (§5.1) and holds the object, in the asker's order
   (holdings §5.4), then the origin.

An asker **MUST NOT** send a holder a `want` for an object that holder's standing record does
not hold. So a holder learns of an asker only that it wants objects the holder has itself told
its neighbours it holds (§9).

### 5.3 The answer

The asker asks the holders of the route in order, one at a time, and stops at the first that
serves bytes it takes:

- **bytes that verify** under the object receipt and the group receipt it holds (jwp-receipts
  §3 steps 1 to 5) are **taken**: `delivered-verified`, whoever sent them (holdings §6). Only
  then does the asker use them, store them, serve them on or acknowledge them;
- **bytes that fail** are refused with jwp-receipts §3's reason and step, `content_hash_mismatch`
  at step 4 for bytes that are not the object's. The asker discards them, and §5.4 applies;
- `miss`, a refusal of §4.4, and a failure of the link move it to the next holder.

When no holder is left it fetches the object from the origin. What comes from the origin is
verified as every object is; this text adds nothing for it.

### 5.4 What an asker sets aside

- **A holder that served bytes that failed** is asked nothing more for the session, whatever
  its records say (`peer_bytes_refused`): every record of its that stands is set aside.
- **A record that overstated**: a holder that answers `miss` with code 1 or 2 for an object its
  standing record holds has stated what it does not hold, and each of its records that holds
  that object is set aside for the session (`holdings_overstated`). A later record of that
  holder is another statement, and is judged as §5.1 judges any record.
- **A record that states a group as another size than its receipt** is already refused whole
  (holdings §5.2, `holdings_leaf_count_mismatch`); its holder is in no route.
- `miss` with code 3 (`peer_declined`), an object past the bound and a link that failed set
  nothing aside.

Nothing here scores a holder beyond the session, and nothing is told to anyone: what an asker
sets aside is its own state (holdings §9).

## 6. The acknowledgement

### 6.1 What the asker signs

For each holder it took objects from, an asker signs, at the end of a window, one
acknowledgement (settlement §3) of exactly the objects that holder served and the asker
verified in the window:

- `mode` 1 and `role` 2: an acknowledgement, by a sink;
- `upstream` the key of the record the asker found the object under
  ([cache-authority](cache-authority.md) §4.5), and `downstream` the asker's key;
- the window, and `prev` the digest of its acknowledgement to that holder before this one
  (settlement §3.7), zero for the first;
- one entry, of class 2 (§6.2), `grant` and `proof` zero, its items the objects taken from that
  holder in the order they were taken, each whole: `start` 0 and `end` the object's length.

An acknowledgement sent over a link names at most 256 objects; an asker with more closes the
window sooner. Where it took objects from several holders it signs one acknowledgement for
each. It sends each to its holder in an `ack` frame with the entry's item list, or as §4.5's
`POST`, and keeps it for the party it settles with.

An object is acknowledged only after it was taken (§5.3). Bytes that failed, a reply that was
not whole and an object fetched from the origin are in no acknowledgement to a holder.

### 6.2 The class, and whose word

An object a holder serves in answer to a `want` is served from what it holds in answer to a
request, so its entry is of class 2, `fetch` (settlement §3.3), whatever the object is: a
retrieval class. This revision's exchange carries no proof of possession of a grant
([live-rooms](live-rooms.md) §10.5), so its entries name no grant: each is a retrieval no grant
was presented for, acknowledged, counted in a statement's lines and classes, and free under a
statement's tariff (settlement §6.2, §7.2). A deployment that pays its peers pays them on the
acknowledged bytes, outside the statement.

A payer honours an acknowledgement only from a key it can place (settlement §13). Outside a
room no grant names the asker, so a payer takes an asker's word when its key is in a population
the payer stated (cache-authority §8.1), and reports any other acknowledgement apart, paid as
no one's. The holder's key is placed as cache-authority §4.4 places any server key.

### 6.3 What a holder checks

A holder takes an acknowledgement sent to it when, in this order:

1. the record reads and verifies (settlement §3.6), with that section's reasons;
2. it is of version 1, `mode` 1 and `role` 2, with one entry, of class 2, whose `grant` and
   `proof` are zero: otherwise `peer_ack_invalid`;
3. its `upstream` is the holder's key: otherwise `ack_wrong_party`;
4. the item list reads and recomputes against the entry (settlement §3.6): `items_invalid`,
   `items_mismatch`;
5. each item is an object the holder served over that link and holds no acknowledgement of,
   whole: otherwise `ack_not_served`.

It answers `ack-result` with the record's digest and 0 when it takes it, 1 when it does not. An
asker may acknowledge fewer objects than it was served: what it did not verify is not its to
sign for, and what a holder served that nobody acknowledged it may state in a declaration
(settlement §4).

## 7. In a page

A sink that is a page of a browser uses what the browser gives it:

- **A data channel** (RFC 8831) is a link. The channel is reliable and ordered (RFC 8832 §5.1,
  `DATA_CHANNEL_RELIABLE`), its label `wai-peer` and its protocol `wai-peer-1`; each frame is
  one binary message.
- **Addresses.** A page gathers the candidates its browser offers under the browser's own
  policy (RFC 8828 §5.2). With the policy's `relay_only` on, it sets its connection to use
  relayed candidates alone (the W3C text's transport policy `relay`), and opens no link where
  it has no relay.
- **An HTTP link** is `fetch`, with credentials omitted, no referrer, and the browser's HTTP
  cache bypassed. A page reaches a holder on a local network only where its browser lets it:
  the *Local Network Access* draft has a browser ask the person before a page of a public
  address makes such a request (its §2.3).
- **A store.** A page that serves keeps objects it has verified, and only those, in a cache of
  the Cache API under its own origin, and serves from it. A cache's entries do not expire by
  themselves; a page removes an object when it stops listing it.
- **Keys.** A page signs with a key it makes for the session and keeps in memory. A page that
  serves and is to be placed signs with the key its operator lists (cache-authority §4.5).

## 8. Reasons

The codes this text adds, each with where it is raised:

| reason | raised at |
|---|---|
| `peer_not_permitted` | §3.3, §5.2 |
| `peer_frame_malformed` | §4.2 |
| `peer_frame_unexpected` | §4.3, §4.4 |
| `peer_object_too_large` | §4.4, §4.5 |
| `peer_link_failed` | §4.4, §4.5 |
| `peer_declined` | §5.4 |
| `peer_receipt_not_held` | §5.2 |
| `holdings_overstated` | §5.4 |
| `peer_bytes_refused` | §5.4 |
| `peer_ack_invalid` | §6.3 |
| `ack_not_served` | §6.3 |

An implementation also raises every code holdings §7 lists, for a record; jwp-receipts §3's
reasons, for a receipt and for bytes that fail one; and, for an acknowledgement, settlement
§11's `ack_invalid`, `self_acknowledged`, `items_invalid`, `items_mismatch` and
`ack_wrong_party`, with `truncated`, `malformed`, `unknown_version`, `bad_key` and
`bad_signature`.

## 9. Security and privacy considerations

**Peers are not trusted.** Nothing a peer says is evidence of an object's bytes. An asker asks
only for objects whose receipts it holds, takes nothing before it verifies, and a holder that
sends other bytes gave it one wasted exchange and is asked nothing more. A holder's record that
overstates costs one exchange and is set aside. A holder cannot make an asker hold more than
`max_object_bytes` of one reply, nor a byte more than a head stated.

**What a holder learns of an asker**, and what it does not:

- **An address.** Over a data channel each end learns the addresses the other's browser offers
  as candidates: under RFC 8828's mode 2, which it recommends where the person has consented
  to no more, those of the route its HTTP traffic takes and its local addresses on that
  interface, which a browser may give as names that resolve only on the local network
  (`draft-ietf-mmusic-mdns-ice-candidates`). With `relay_only` each end learns the relay's
  address alone, and the relay learns both. Over an HTTP link the holder learns the address the
  request came from, as any server does, and, from a page, the origin of the page that made
  it. An address places a viewer roughly and may identify a household.
- **What was asked for.** Each `want` names a group and a position: a holder learns that this
  asker wants that object now, and so what it is watching and where in it, for the objects the
  holder itself listed and for no other (§5.2).
- **A key.** An acknowledgement names the asker's key, its window and the objects it took. A
  key made for the session links nothing to another session; an asker **MAY** acknowledge
  nothing, and its holder then states the delivery itself (settlement §4).
- **When.** The times of its requests.
- **Nothing else.** The exchange carries no name, account, grant, cookie or credential of the
  viewer, no list of what else it wants or holds, and nothing of what it fetched from the
  origin or from another holder.

**What an asker learns of a holder**: its address, as above; its record, which says what it
holds (holdings §9); and whether it served.

**What the service learns**: whatever introduces two sinks learns that it did; the origin sees
no request for an object a peer served, and learns of it from the acknowledgements, where the
asker hands them on.

**Whose choice.** Joining is the sink's (§3): a viewer who has not turned `ask` on has its
address shown to no peer and names no object to one, and a viewer who has not turned `serve` on
tells no neighbour what it holds. Turning a setting off stops the next exchange; what a peer
already learned it keeps. Where a regime asks the operator to state the lawful basis on which
it processes what an acknowledgement reveals, the operator states it as settlement §13 says.

**A holder's key on the link.** A link is not authenticated by this exchange. A party that
answers on a link in a holder's place can serve only bytes that verify, and the acknowledgement
names the record's key, so what it gains is to have served for another's credit. One that
sends other bytes in a holder's place has that holder set aside by that asker for its session:
it costs the asker a route and the holder that asker's requests, and nothing else.

**Bounded work.** One request outstanding per link; a reply of at most `max_object_bytes`; a
record of at most 1,048,576 bytes; an acknowledgement of at most 256 items; a verification of
one hash and one inclusion proof per object.

## 10. Conformance

[`live-conformance/peer-delivery/`](../live-conformance/peer-delivery/) holds the corpus,
written from fixed seeds by `wai_peer_delivery_vectors gen` as `vectors.json`. It has:

- **A world**: two groups with their receipts, their objects' receipts and envelopes, one object
  past one `data` frame.
- **Frames**: each frame of §4.2 with what it reads to, and a control for each bound of its
  table.
- **Transfers**: a head held to what was asked and to the bound, and replies in one frame and
  two, a byte long, a frame short and a frame past the end.
- **Gates**: every setting of the four switches of §3.
- **Sessions**: an asker's rounds, each with the records offered, what each holder's link
  answers, each wanted object's route, attempts and source, and the acknowledgements signed. A
  group taken whole from one peer; a peer that serves altered bytes, refused, set aside and
  named by no acknowledgement; a record that overstates a group's size, never asked; a holder
  that lists an object it does not hold, set aside, and its later record judged anew; two
  windows chained; the viewer's setting off and the operator's policy off; a holder that
  declines and a link that fails; an object with no receipt held; an object past the bound; the
  asker's own record; an expired record; and two holders acknowledged apart.
- **Answers** and **acknowledgements at a holder**: each answer of §4.3, and each check of §6.3
  with a control.

A conforming implementation gives every case the outcome the corpus states.
`wai_peer_delivery_vectors verify` and the test `tests/peer_delivery.rs` check that the file
regenerates byte for byte, that `wai-rs` gives every case its outcome, that every reason of §8
appears, and that §8 lists exactly the reference's codes.
`live-conformance/peer_delivery_verify.py` is the second implementation, written from this
text: it shares no code with `wai-rs`; it takes records, receipts and acknowledgements from the
second implementations of the texts that define them (`holdings_verify.py`,
`jwp-conformance/jwp_verify.py`, `settlement_verify.py`), writes the frames, the gate, the
route, the session and the holder's checks itself, rebuilds each acknowledgement's bytes before
its signature and verifies the signature, mutates copies of the corpus and requires every copy
to fail, and drops each of its own checks in turn and requires the corpus to catch the weakened
verifier.

In a page, `wai-web/peer/` holds the client and two runs of it over the corpus's world.
`verify_peer.mjs` runs an asker and its holders in one process over links in memory and over a
local HTTP link, with the faults above planted. `run.mjs` opens pages in a browser: holders
that serve over data channels, one of them altering bytes and one listing what it does not
hold, a local HTTP holder, an asker with both switches on and one with its setting off; it
requires every object the asker shows to have verified, the altering peer to be named by no
acknowledgement, every acknowledgement to be taken by its holder, the origin to have served
only what no peer did, and the page with its setting off to have opened no connection.