open-standards/wai/extensions/peer-delivery.md

WAI Extension: Peer Delivery

Status: Draft. Registers no capability and no record. It is the exchange by which a sink fetches objects from the neighbours that hold them before it spends the backhaul, over the records other texts define: a neighbour's holdings record says where to ask (holdings §5.4), the publisher's receipts say what the answer must be (jwp-receipts §3), and the asker's settlement acknowledgement says what was served (settlement §3). A sink joins by its own choice: asking a peer and serving one are each off unless the operator's policy and the viewer's setting both allow it (§3). The exchange is eight frames over a link that keeps message boundaries, a WebRTC data channel among them, and three requests over HTTP for a neighbour reached that way (§4). An asker asks only for an object whose object receipt it already holds, takes nothing a peer sent before the bytes verify under that receipt, sets aside a holder that served other bytes and a record that stated what its holder did not hold, and goes to the origin for what is left (§5). It signs an acknowledgement for what it verified, by holder and by window, so what each peer served recomputes from signed records (§6). §9 says what a peer learns of a viewer. WAI defines no transport. The reference implementation is the peer_delivery module of wai-rs (feature peer_delivery) and, in a page, the peer bundle of wai-web; the corpus is live-conformance/peer-delivery/, and its second implementation is live-conformance/peer_delivery_verify.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

Texts read for this revision: the WAI texts this one cites; RFC 8831 (§6.1, §6.6: a data channel's reliability and its message size) and RFC 8832 (§5.1: the channel's type, label and protocol); RFC 8828 (§3, §5.1, §5.2: which addresses a WebRTC endpoint gathers, and its four modes); RFC 8826 §4.2.4; draft-ietf-mmusic-mdns-ice-candidates-03 §1 and §3.1.1; RFC 7574 §3.2 and §5; RFC 9110 §15.5.5, §15.5.21 and §15.6.4; the W3C WebRTC: Real-Time Communication in Browsers Recommendation (a data channel's label, protocol, order and largest message, and the transport policy relay of a connection's configuration); the W3C Service Workers text, §5.4 (Cache); and the Web Incubator Community Group's draft Local Network Access (§1, §2.1 to §2.3). A clause number of one of them in this text is that text's.

1. Scope

In scope: who may ask and who may serve (§3); the exchange between an asker and a holder, over a link that keeps message boundaries and over HTTP (§4); what an asker does with its neighbours' records and with each answer (§5); the acknowledgement of what a peer served, its object class and what a holder checks of it (§6); what a page uses of its browser (§7); and what each party learns (§9).

Out of scope: how two sinks are introduced to each other and how a link between them is set up, which is the deployment's; how holdings records reach an asker before it has a link (holdings §1); how a node chooses what to keep; a price; and a record of service beside settlement's acknowledgement, which this text does not add.

Relationship to the core spec: this text defines no capability, no payload format and no record (SPEC §5). Every signed byte it relies on is another text's.

2. Terms

• Asker, holder, group, link hash, position: as holdings §2. A peer is a holder that is a sink too; what this text says of a holder holds for a neighbourhood cache as it does for a peer.
• Origin: where the asker fetches an object when it asks no holder for it, or none serves it: the service it takes delivery from.
• Link: a path between an asker and one holder that carries messages whole and in order, each way, with none lost. An HTTP link is a base URL at which a holder answers the requests of §4.5.
• Session: one asker's exchanges from when it starts until it stops. What an asker sets aside (§5.4) it sets aside for its session.
• Instant: milliseconds, as holdings §2. A window is settlement's, in microseconds (settlement §2).
• Integers in frames are unsigned and big-endian, as a holdings record's are.

3. Joining

3.1 The operator's policy

The operator of the service a sink takes delivery from states, in the configuration it gives its sinks, four values:

value	meaning	where the operator states nothing
ask	its sinks may fetch from peers	no
serve	its sinks may serve peers from what they hold	no
max_object_bytes	the most bytes of one object a sink takes from a peer	4,194,304
relay_only	its sinks reach peers only through a relay (§9)	no

How the configuration reaches a sink is the deployment's. A sink with no configuration has an operator that states nothing.

3.2 The viewer's setting

A sink keeps two settings of its own, ask and serve, each off until the person using the sink turns it on, and each theirs to turn off again. A sink MUST NOT turn either on by itself, and MUST show, where it offers them, what §9 says a peer learns.

3.3 The rule

A sink asks only while the policy's ask and its own ask are both on, and serves only while the policy's serve and its own serve are both on. Asking is not serving: either may be on without the other.

• A sink that may not ask opens no link to a holder, sends no frame and no request of §4.5, takes no object from a peer and signs no acknowledgement to one. Every object it wants is the origin's (peer_not_permitted).
• A sink that may not serve signs no holdings record for its neighbours, accepts no link, and answers a want it is sent, on a link it already has, with miss and code 3.
• A setting turned off takes effect at the next object: an exchange under way may finish or be dropped, and nothing new starts.

4. The exchange

4.1 The link and the version

The exchange is version 1. An asker opens it with record-want, which states the version; a holder that does not speak that version closes the link. Each frame is one message of the link. On a link, an asker has at most one request outstanding: it sends record-want, want or ack and waits for the whole reply before it sends another.

4.2 Frames

A frame is one byte of type, then its body:

type	name	from	body
1	record-want	asker	1 byte: the version, 1
2	record-head	holder	len, 8 bytes: the bytes of the holder's record, 1 to 1,048,576, which follow in data frames
3	want	asker	group, 32 bytes, and index, 8 bytes: object index of the group with that link hash
4	head	holder	group, index, and len, 8 bytes, at least 1: the object's bytes, which follow in data frames
5	data	holder	1 to 16,384 bytes
6	miss	holder	group, index, and code, 1 byte: 1 the holder does not hold the object now, 2 its record does not list it, 3 it will not serve now
7	ack	asker	record_len, 4 bytes, at least 1; an acknowledgement of that many bytes (§6.1); then its item list, 1 to 256 items of 48 bytes
8	ack-result	holder	digest, 32 bytes, the acknowledgement's (settlement §3.5), and 1 byte: 0 taken, 1 not taken

A reader reads exactly the bytes of one message as one frame and refuses it peer_frame_malformed when it is empty, its type is not 1 to 8, its body is not of its type's length, or a field is outside the range the table gives. A record-want of another version is peer_frame_malformed in this revision. An object's bytes travel in frames of at most 16,384 bytes because RFC 8831 §6.6 asks a sender that cannot interleave messages to keep each within 16 KB.

4.3 The holder

A holder answers record-want with record-head and the bytes of the one record of its own that stands (holdings §5.3), and want with the first of these that holds:

1. miss, code 3, when it may not serve (§3.3) or will not serve now;
2. miss, code 2, when that record does not hold the object (holdings §5.4): a holder serves over this exchange only what it has told its neighbours it holds;
3. miss, code 1, when it does not hold the object's bytes now;
4. head with the object's length, then its bytes.

It answers ack as §6.3 says. A holder MUST NOT send bytes it has not itself verified under the object's receipt, and learns nothing from doing so: its asker verifies them again (§5.3). A frame a holder is sent that is not record-want, want or ack, or that arrives before its reply to the one before is whole, is peer_frame_unexpected, and it closes the link.

4.4 A reply

The bytes a record-head or a head announces follow in data frames, each of 16,384 bytes but the last, which holds what remains. An asker holds a reply to what it asked:

• a head whose group or index is not what it asked for, a data frame that is not the next 16,384 bytes or the last bytes where fewer remain, a frame after the reply is whole, and any frame of another type than its request admits: peer_frame_unexpected;
• a head whose len is above max_object_bytes: peer_object_too_large. The asker takes none of its bytes;
• a link that closes, or gives no frame within the time the asker allows, before the reply is whole: peer_link_failed.

After any of the three the asker closes the link and asks that holder nothing more over it. A reply is whole when its bytes are exactly len; an asker never holds more of one than its head stated, and it stores nothing of a reply that is not whole.

4.5 Over HTTP

A holder reached by HTTP answers, under its base URL:

request	answer
GET holdings	200 with the record's bytes
GET object/<group>/<index>	200 with the object's bytes, whose length the response states; 404 for miss code 1 or 2; 503 for code 3
POST ack, its body an ack frame's body	200 with the acknowledgement's digest when it is taken; 422 when it is not

<group> is the link hash as 64 lowercase hexadecimal digits and <index> the position in decimal with no leading zero. The rules of §4.3 and §4.4 hold with a response for a reply: a stated length above max_object_bytes, or a body that reaches it with no length stated, is peer_object_too_large, and the asker reads no further; any other status, a body shorter or longer than its stated length, and a request that fails are peer_link_failed. A 404 is read as code 1. An asker's requests carry no credentials, no cookie and no referrer (§7).

5. The asker

5.1 Its records

An asker holds its neighbours' records in the order it prefers them (holdings §5.4) and judges them at an instant as holdings §5.1 to §5.3 do, against the group receipts it holds: a record that is refused, superseded or set aside there takes no part. Of those that stand, it sets aside the ones §5.4 names. Its own record, where it has one, is in no route: an asker does not ask itself.

5.2 The route

For a wanted object, the asker's route is, at an instant:

1. the origin alone, peer_not_permitted, when it may not ask (§3.3);
2. the origin alone, peer_receipt_not_held, when it does not hold the object's object receipt and the group receipt it is included under, both verified (jwp-receipts §3 steps 1, 2 and 5): it could hold nothing a peer sent for it to a signed digest, so it asks no peer for it;
3. otherwise the holders whose record stands (§5.1) and holds the object, in the asker's order (holdings §5.4), then the origin.

An asker MUST NOT send a holder a want for an object that holder's standing record does not hold. So a holder learns of an asker only that it wants objects the holder has itself told its neighbours it holds (§9).

5.3 The answer

The asker asks the holders of the route in order, one at a time, and stops at the first that serves bytes it takes:

• bytes that verify under the object receipt and the group receipt it holds (jwp-receipts §3 steps 1 to 5) are taken: delivered-verified, whoever sent them (holdings §6). Only then does the asker use them, store them, serve them on or acknowledge them;
• bytes that fail are refused with jwp-receipts §3's reason and step, content_hash_mismatch at step 4 for bytes that are not the object's. The asker discards them, and §5.4 applies;
• miss, a refusal of §4.4, and a failure of the link move it to the next holder.

When no holder is left it fetches the object from the origin. What comes from the origin is verified as every object is; this text adds nothing for it.

5.4 What an asker sets aside

• A holder that served bytes that failed is asked nothing more for the session, whatever its records say (peer_bytes_refused): every record of its that stands is set aside.
• A record that overstated: a holder that answers miss with code 1 or 2 for an object its standing record holds has stated what it does not hold, and each of its records that holds that object is set aside for the session (holdings_overstated). A later record of that holder is another statement, and is judged as §5.1 judges any record.
• A record that states a group as another size than its receipt is already refused whole (holdings §5.2, holdings_leaf_count_mismatch); its holder is in no route.
• miss with code 3 (peer_declined), an object past the bound and a link that failed set nothing aside.

Nothing here scores a holder beyond the session, and nothing is told to anyone: what an asker sets aside is its own state (holdings §9).

6. The acknowledgement

6.1 What the asker signs

For each holder it took objects from, an asker signs, at the end of a window, one acknowledgement (settlement §3) of exactly the objects that holder served and the asker verified in the window:

• mode 1 and role 2: an acknowledgement, by a sink;
• upstream the key of the record the asker found the object under (cache-authority §4.5), and downstream the asker's key;
• the window, and prev the digest of its acknowledgement to that holder before this one (settlement §3.7), zero for the first;
• one entry, of class 2 (§6.2), grant and proof zero, its items the objects taken from that holder in the order they were taken, each whole: start 0 and end the object's length.

An acknowledgement sent over a link names at most 256 objects; an asker with more closes the window sooner. Where it took objects from several holders it signs one acknowledgement for each. It sends each to its holder in an ack frame with the entry's item list, or as §4.5's POST, and keeps it for the party it settles with.

An object is acknowledged only after it was taken (§5.3). Bytes that failed, a reply that was not whole and an object fetched from the origin are in no acknowledgement to a holder.

6.2 The class, and whose word

An object a holder serves in answer to a want is served from what it holds in answer to a request, so its entry is of class 2, fetch (settlement §3.3), whatever the object is: a retrieval class. This revision's exchange carries no proof of possession of a grant (live-rooms §10.5), so its entries name no grant: each is a retrieval no grant was presented for, acknowledged, counted in a statement's lines and classes, and free under a statement's tariff (settlement §6.2, §7.2). A deployment that pays its peers pays them on the acknowledged bytes, outside the statement.

A payer honours an acknowledgement only from a key it can place (settlement §13). Outside a room no grant names the asker, so a payer takes an asker's word when its key is in a population the payer stated (cache-authority §8.1), and reports any other acknowledgement apart, paid as no one's. The holder's key is placed as cache-authority §4.4 places any server key.

6.3 What a holder checks

A holder takes an acknowledgement sent to it when, in this order:

1. the record reads and verifies (settlement §3.6), with that section's reasons;
2. it is of version 1, mode 1 and role 2, with one entry, of class 2, whose grant and proof are zero: otherwise peer_ack_invalid;
3. its upstream is the holder's key: otherwise ack_wrong_party;
4. the item list reads and recomputes against the entry (settlement §3.6): items_invalid, items_mismatch;
5. each item is an object the holder served over that link and holds no acknowledgement of, whole: otherwise ack_not_served.

It answers ack-result with the record's digest and 0 when it takes it, 1 when it does not. An asker may acknowledge fewer objects than it was served: what it did not verify is not its to sign for, and what a holder served that nobody acknowledged it may state in a declaration (settlement §4).

7. In a page

A sink that is a page of a browser uses what the browser gives it:

• A data channel (RFC 8831) is a link. The channel is reliable and ordered (RFC 8832 §5.1, DATA_CHANNEL_RELIABLE), its label wai-peer and its protocol wai-peer-1; each frame is one binary message.
• Addresses. A page gathers the candidates its browser offers under the browser's own policy (RFC 8828 §5.2). With the policy's relay_only on, it sets its connection to use relayed candidates alone (the W3C text's transport policy relay), and opens no link where it has no relay.
• An HTTP link is fetch, with credentials omitted, no referrer, and the browser's HTTP cache bypassed. A page reaches a holder on a local network only where its browser lets it: the Local Network Access draft has a browser ask the person before a page of a public address makes such a request (its §2.3).
• A store. A page that serves keeps objects it has verified, and only those, in a cache of the Cache API under its own origin, and serves from it. A cache's entries do not expire by themselves; a page removes an object when it stops listing it.
• Keys. A page signs with a key it makes for the session and keeps in memory. A page that serves and is to be placed signs with the key its operator lists (cache-authority §4.5).

8. Reasons

The codes this text adds, each with where it is raised:

reason	raised at
peer_not_permitted	§3.3, §5.2
peer_frame_malformed	§4.2
peer_frame_unexpected	§4.3, §4.4
peer_object_too_large	§4.4, §4.5
peer_link_failed	§4.4, §4.5
peer_declined	§5.4
peer_receipt_not_held	§5.2
holdings_overstated	§5.4
peer_bytes_refused	§5.4
peer_ack_invalid	§6.3
ack_not_served	§6.3

An implementation also raises every code holdings §7 lists, for a record; jwp-receipts §3's reasons, for a receipt and for bytes that fail one; and, for an acknowledgement, settlement §11's ack_invalid, self_acknowledged, items_invalid, items_mismatch and ack_wrong_party, with truncated, malformed, unknown_version, bad_key and bad_signature.

9. Security and privacy considerations

Peers are not trusted. Nothing a peer says is evidence of an object's bytes. An asker asks only for objects whose receipts it holds, takes nothing before it verifies, and a holder that sends other bytes gave it one wasted exchange and is asked nothing more. A holder's record that overstates costs one exchange and is set aside. A holder cannot make an asker hold more than max_object_bytes of one reply, nor a byte more than a head stated.

What a holder learns of an asker, and what it does not:

• An address. Over a data channel each end learns the addresses the other's browser offers as candidates: under RFC 8828's mode 2, which it recommends where the person has consented to no more, those of the route its HTTP traffic takes and its local addresses on that interface, which a browser may give as names that resolve only on the local network (draft-ietf-mmusic-mdns-ice-candidates). With relay_only each end learns the relay's address alone, and the relay learns both. Over an HTTP link the holder learns the address the request came from, as any server does, and, from a page, the origin of the page that made it. An address places a viewer roughly and may identify a household.
• What was asked for. Each want names a group and a position: a holder learns that this asker wants that object now, and so what it is watching and where in it, for the objects the holder itself listed and for no other (§5.2).
• A key. An acknowledgement names the asker's key, its window and the objects it took. A key made for the session links nothing to another session; an asker MAY acknowledge nothing, and its holder then states the delivery itself (settlement §4).
• When. The times of its requests.
• Nothing else. The exchange carries no name, account, grant, cookie or credential of the viewer, no list of what else it wants or holds, and nothing of what it fetched from the origin or from another holder.

What an asker learns of a holder: its address, as above; its record, which says what it holds (holdings §9); and whether it served.

What the service learns: whatever introduces two sinks learns that it did; the origin sees no request for an object a peer served, and learns of it from the acknowledgements, where the asker hands them on.

Whose choice. Joining is the sink's (§3): a viewer who has not turned ask on has its address shown to no peer and names no object to one, and a viewer who has not turned serve on tells no neighbour what it holds. Turning a setting off stops the next exchange; what a peer already learned it keeps. Where a regime asks the operator to state the lawful basis on which it processes what an acknowledgement reveals, the operator states it as settlement §13 says.

A holder's key on the link. A link is not authenticated by this exchange. A party that answers on a link in a holder's place can serve only bytes that verify, and the acknowledgement names the record's key, so what it gains is to have served for another's credit. One that sends other bytes in a holder's place has that holder set aside by that asker for its session: it costs the asker a route and the holder that asker's requests, and nothing else.

Bounded work. One request outstanding per link; a reply of at most max_object_bytes; a record of at most 1,048,576 bytes; an acknowledgement of at most 256 items; a verification of one hash and one inclusion proof per object.

10. Conformance

live-conformance/peer-delivery/ holds the corpus, written from fixed seeds by wai_peer_delivery_vectors gen as vectors.json. It has:

• A world: two groups with their receipts, their objects' receipts and envelopes, one object past one data frame.
• Frames: each frame of §4.2 with what it reads to, and a control for each bound of its table.
• Transfers: a head held to what was asked and to the bound, and replies in one frame and two, a byte long, a frame short and a frame past the end.
• Gates: every setting of the four switches of §3.
• Sessions: an asker's rounds, each with the records offered, what each holder's link answers, each wanted object's route, attempts and source, and the acknowledgements signed. A group taken whole from one peer; a peer that serves altered bytes, refused, set aside and named by no acknowledgement; a record that overstates a group's size, never asked; a holder that lists an object it does not hold, set aside, and its later record judged anew; two windows chained; the viewer's setting off and the operator's policy off; a holder that declines and a link that fails; an object with no receipt held; an object past the bound; the asker's own record; an expired record; and two holders acknowledged apart.
• Answers and acknowledgements at a holder: each answer of §4.3, and each check of §6.3 with a control.

A conforming implementation gives every case the outcome the corpus states. wai_peer_delivery_vectors verify and the test tests/peer_delivery.rs check that the file regenerates byte for byte, that wai-rs gives every case its outcome, that every reason of §8 appears, and that §8 lists exactly the reference's codes. live-conformance/peer_delivery_verify.py is the second implementation, written from this text: it shares no code with wai-rs; it takes records, receipts and acknowledgements from the second implementations of the texts that define them (holdings_verify.py, jwp-conformance/jwp_verify.py, settlement_verify.py), writes the frames, the gate, the route, the session and the holder's checks itself, rebuilds each acknowledgement's bytes before its signature and verifies the signature, mutates copies of the corpus and requires every copy to fail, and drops each of its own checks in turn and requires the corpus to catch the weakened verifier.

In a page, wai-web/peer/ holds the client and two runs of it over the corpus's world. verify_peer.mjs runs an asker and its holders in one process over links in memory and over a local HTTP link, with the faults above planted. run.mjs opens pages in a browser: holders that serve over data channels, one of them altering bytes and one listing what it does not hold, a local HTTP holder, an asker with both switches on and one with its setting off; it requires every object the asker shows to have verified, the altering peer to be named by no acknowledgement, every acknowledgement to be taken by its holder, the origin to have served only what no peer did, and the page with its setting off to have opened no connection.