C3's synthesis subject `samples` (a picture a party presents and does not publish) met takedown's exhaustive match once both slices were in one tree (E0004). Such samples are no item a holder serves or keeps for others, so a closure claim whose subject they are adds no subject and needs no purge: takedown §4.2 rule 3, the reference and its second implementation say so, and the corpus's shared world gains such a claim, so every case exercises it. Co-Authored-By: Claude Opus 5.5 <[email protected]>
721 lines
44 KiB
Markdown
721 lines
44 KiB
Markdown
# WAI Extension: Takedown
|
||
|
||
> Status: Draft. Registers the record `wai.media.takedown`: the removal of an item and of
|
||
> everything WAI derived from it, from every party that held it, timed against the clock of
|
||
> the regime the service runs under. A service signs a **takedown** of a source (an object by
|
||
> its content hash, a group, a track or a take) in answer to a signed **notice** or **order**,
|
||
> or of its own initiative. The takedown covers the source's **derivation closure**: every
|
||
> object, group and track that a derivation record ([deriving-edge](deriving-edge.md)) or an
|
||
> origin claim ([synthesis](synthesis.md)) derives from it, transitively — transcodes, ladder
|
||
> rungs, crops and thumbnails, upscales, re-packages, generated tracks — which a verifier
|
||
> given the records computes exactly and holds the takedown's stated set to. A copy made
|
||
> outside every WAI derivation is not in the closure; a **perceptual match** may be added
|
||
> beside it, as a separately labelled class that is never counted as exact. Each party that
|
||
> held a covered object (a store the service names, an edge that derived from it, a party
|
||
> that claimed to, a publisher of a covered group, and a party that served or received it as
|
||
> a [settlement](settlement.md) acknowledgement shows) signs a **purge**: the subjects it no
|
||
> longer holds or serves, and when. From the clock's start to the last purge required the
|
||
> verifier computes the **time to removal** and checks it against the clock the policy names,
|
||
> read from a **registry of regime clocks** by identifier and duration; a holder's purge that
|
||
> never came is `purge_missing`, a removal past the deadline is `deadline_missed`, and a
|
||
> removal inside a reply window is `window_not_observed`. A service's signed **counts**, per
|
||
> clock and per category, are recomputed from its takedowns. WAI decides no legal question:
|
||
> it computes what was removed, where and when. No new transport. The reference
|
||
> implementation is the `takedown` module of `wai-rs` (feature `takedown`), and the reference
|
||
> relay's DVR store honours a signed takedown; the corpus is
|
||
> [`live-conformance/takedown/`](../live-conformance/takedown/), and its second implementation
|
||
> is `live-conformance/takedown_verify.py`.
|
||
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.
|
||
>
|
||
> Texts read for this revision, each pinned to the version named: Public Law 119-12 (139 Stat.
|
||
> 55), section 3(a)(1), 3(a)(3) and 3(b), as enrolled; the Information Technology
|
||
> (Intermediary Guidelines and Digital Media Ethics Code) Rules, 2021, rule 3(1)(d) and rule
|
||
> 3(2), in the consolidated text "updated as on 10.02.2026" that incorporates G.S.R. 120(E) of
|
||
> 10 February 2026, and the questions and answers published with that notification for the
|
||
> date it came into force, 20 February 2026 (the Gazette notification itself was not read);
|
||
> Regulation (EU) 2024/1083, Article 18 and Article 29, as published in the Official Journal
|
||
> (L series, 17.4.2024); and Regulation (EU) 2022/2065, Article 15(1), Article 16 and Article
|
||
> 93, as the Publications Office serves its consolidated version 02022R2065-20221027. A
|
||
> section number of one of those texts in this extension is that version's. The durations
|
||
> those texts set are not restated here: they are the entries of the registry of §4.
|
||
|
||
## 1. Scope
|
||
|
||
In scope: the subjects a takedown names (§3); the registry of regime clocks (§4); the
|
||
records, their encodings, signatures and rules (§5); the derivation closure, how it is
|
||
computed from derivation records, origin claims and delivery receipts, and what it does not
|
||
reach (§6); who must purge, what a purge covers, and the time to removal against a clock (§7);
|
||
verification and what it states (§8); the signed counts and their recomputation (§9); and what
|
||
a relay, a recording and an HTTP origin do with a takedown (§10).
|
||
|
||
Out of scope: a transport, and how a notice reaches a service or a takedown reaches a holder;
|
||
which regime applies to a notice, whether an item is unlawful, and whether a clock's duration
|
||
was met in law: a service names its clocks, and a verifier computes against the ones named;
|
||
how a moderator or a classifier reaches a verdict; how a perceptual match is computed; the
|
||
reinstatement of a removed object, which this revision does not express; and the restriction
|
||
of accounts or payments.
|
||
|
||
Relationship to the core spec: `wai.media.takedown` is a **record** ([SPEC](../SPEC.md) §5): a
|
||
signed record a WAI tool emits and verifies. It names no payload format and is not
|
||
dispatchable.
|
||
|
||
Relationship to [live-moderation](live-moderation.md): live-moderation decides on the items of
|
||
a room, by `decision` events in the room's chains and `WMS1` statements beside them, each of
|
||
version 1, whose `target_type` admits an event or a media object and nothing else, so that a
|
||
verifier of that revision refuses any other (live-moderation §4.3). A catalogue or on-demand
|
||
object has no room, no room clock and no chain to carry a decision. This extension therefore
|
||
defines a **sibling record** rather than a new version of `WMS1`: a takedown carries the
|
||
statement's fields (its ground, its explanation and facts, whether automated means detected the
|
||
item or took the decision, and a category of live-moderation §9.5) as JSON, signed by the
|
||
service, so that live-moderation's records and verifiers are unchanged. A room item restricted
|
||
by a room decision is live-moderation's; the same object delivered beyond the room, named by
|
||
its content hash, is this extension's too.
|
||
|
||
Relationship to [deriving-edge](deriving-edge.md) and [synthesis](synthesis.md): their records
|
||
are the closure's evidence (§6). This extension reads them (deriving-edge §5.3 and §5.2,
|
||
synthesis §2 and §3.1) and verifies their signatures; it does not re-execute a derivation or
|
||
check a record's delivery receipts, which those extensions do.
|
||
|
||
Relationship to [settlement](settlement.md): an acknowledgement (`WDA1`, settlement §3) signed by
|
||
the party downstream of a delivery names, by content hash, the objects it received from its
|
||
upstream. It is the evidence that a party served a covered object, or, as a hop, received one
|
||
(§7.1). This extension reads acknowledgements and declarations (settlement §3.6, §4) and their
|
||
item lists; it settles nothing.
|
||
|
||
Relationship to [jwp-receipts](jwp-receipts.md): a group's objects are read from its group and
|
||
object receipts (§6.3). The processing of a notice's personal data is the service's own; the
|
||
basis it relies on is declared with a `lawful-basis` claim (jwp-receipts §2.4.6), and nothing
|
||
in this extension asks anyone's consent.
|
||
|
||
## 2. Terms and encodings
|
||
|
||
- **Service**: the party that signs a policy and its takedowns, named by its Ed25519 key (RFC
|
||
8032), the policy's `signer_pubkey`.
|
||
- **Holder**: a party that held a covered subject (§7.1), named by its key.
|
||
- **Subject**: what a takedown, a purge or a closure names (§3).
|
||
- **Covered**: in the closure of a takedown (§6).
|
||
- **Time**: an instant written `YYYY-MM-DDTHH:MM:SSZ`, a date-time of jwp-receipts §2.4.3 in
|
||
UTC with whole seconds, a capital `T` and `Z`, a year of 1970 or later, and a second of at
|
||
most 59. Its value is the number of seconds from 1970-01-01T00:00:00Z in the proleptic
|
||
Gregorian calendar, with no leap second; a difference of two times is a number of seconds.
|
||
- **Date**: `YYYY-MM-DD`, a date of RFC 3339 §5.6.
|
||
- **Clock identifier**: 1 to 64 bytes of lowercase ASCII letters and digits in groups joined
|
||
by single hyphens (`[a-z0-9]+(-[a-z0-9]+)*`).
|
||
- **Text**: a JSON string of at most the number of code points its member states.
|
||
|
||
Every record is a JSON object (RFC 8259) under the encodings of jwp-receipts §2.4.3: byte
|
||
strings in lowercase hexadecimal; integers unsigned, at most 2^53 − 1, in plain decimal; no
|
||
other number and no boolean. A record is read as jwp-receipts §3.1 step 1 reads a claim: RFC
|
||
8259's grammar, no member named twice in one object, at most 32 levels of nesting. Each record
|
||
has exactly the members its section lists, at every level, and the `kind` that names it; one
|
||
that does not, or whose member breaks its rule, is `record_malformed`.
|
||
|
||
Every record but the registry carries `signer_pubkey` (32 bytes) and `sig` (64 bytes):
|
||
|
||
```
|
||
sig = Ed25519(signer_key, "wai:takedown\x01" || JCS(record without "sig"))
|
||
link_hash = BLAKE3("wai:takedown-link\x01" || JCS(record))
|
||
```
|
||
|
||
JCS is RFC 8785, which on these values reduces to what jwp-receipts §2.4.3 states. The
|
||
signature is verified strictly ([SPEC](../SPEC.md) §7 *Signatures*): a key that fails step 1
|
||
or 2 is `bad_key`, and a signature that fails a later step `bad_signature`. A record is named
|
||
by its link hash wherever another names it; the registry, which carries no signature, is named
|
||
by its link hash too.
|
||
|
||
## 3. Subjects
|
||
|
||
A subject is one of four objects:
|
||
|
||
| `type` | members | what it names |
|
||
|---|---|---|
|
||
| `object` | `content_hash` (32 B hex) | an object by its content hash (jwp-receipts §2.1): every copy of those exact bytes, wherever it is held |
|
||
| `group` | `receipt` (32 B hex) | a group by the link hash of its group receipt (jwp-receipts §3 step 3) |
|
||
| `track` | `track` (32 B hex), `from_group` (integer) | a track by its digest (live-rooms §6.2: SHA-256 over the rendering of its namespace, one zero byte and its name), from group `from_group` on, as synthesis §3.1 names one |
|
||
| `take` | `take` (32 B hex) | a take by its take id ([capture-binding](capture-binding.md) §3.2): its chunks, wherever they are held by take |
|
||
|
||
The first three are the subjects of synthesis §3.1, member for member. A list of subjects is
|
||
in ascending order of their JCS forms as bytes, none twice.
|
||
|
||
An object's content hash names its bytes, not a place: two parties that hold the same envelope
|
||
hold one subject, and an identical copy published again as a new object of another group has
|
||
the same content hash and is covered by the same subject.
|
||
|
||
## 4. The registry of regime clocks
|
||
|
||
A **clock** is a duration a regime sets for a removal or before one, by identifier. Clocks are
|
||
data, held in a registry document, never fixed in this text or in an implementation; a policy
|
||
pins a registry by its link hash (§5.1), and a verifier reads every duration from the registry
|
||
it is given.
|
||
|
||
The registry, `kind` `"wai-takedown-clocks"`:
|
||
|
||
| member | type | meaning |
|
||
|---|---|---|
|
||
| `kind` | string | `"wai-takedown-clocks"` |
|
||
| `clocks` | array | 1 to 64 clocks, in ascending order of `id`, none twice |
|
||
|
||
A clock:
|
||
|
||
| member | type | meaning |
|
||
|---|---|---|
|
||
| `id` | clock identifier | its name |
|
||
| `type` | string | `deadline`: the removal is due at most `duration_s` after the clock starts; `reply_window`: no removal takes effect before `duration_s` after the clock starts, or before a reply that comes earlier; `unfixed`: the regime sets no duration |
|
||
| `duration_s` | integer or `null` | at least 1 for `deadline` and `reply_window`; `null` for `unfixed` |
|
||
| `starts` | string | `receipt`: the service's receipt of the notice or order; `statement`: the service's communication of its statement of reasons to the party whose item it restricts |
|
||
| `instrument` | text, 1 to 500 | the instrument that sets it |
|
||
| `section` | text, 1 to 200 | the provision of that instrument |
|
||
| `applies_from` | date | the date from which the instrument applies, as the registry's author read it; a verifier states it and decides nothing by it |
|
||
|
||
WAI publishes a registry at [`registry/takedown-clocks.json`](../registry/takedown-clocks.json),
|
||
each entry citing the instrument and the provision it is read from. A service may run under
|
||
another registry; its policy names the one it runs under. A clock is added to a registry, or
|
||
corrected, by a new registry, which has a new link hash, and a policy that adopts it is a new
|
||
policy.
|
||
|
||
## 5. Records
|
||
|
||
### 5.1 The policy
|
||
|
||
A **policy**, `kind` `"wai-takedown-policy"`, signed by the service, says which clocks the
|
||
service runs under and which parties hold what it publishes:
|
||
|
||
| member | type | meaning |
|
||
|---|---|---|
|
||
| `kind` | string | `"wai-takedown-policy"` |
|
||
| `clocks` | 32 B hex | the link hash of the registry it runs under (§4) |
|
||
| `use` | array | 0 to 64 clock identifiers, ascending, none twice: the clocks its takedowns may name |
|
||
| `stores` | array | 1 to 256 stores, in ascending order of `key`, none twice: each `{ "key", "role" }`, a key (32 B hex) and its role, one of `origin`, `store`, `relay`, `cache` and `edge` |
|
||
| `issued_at` | time | when the service signed it |
|
||
| `signer_pubkey`, `sig` | | §2 |
|
||
|
||
A **store** is a party the service operates or engages that holds what the service publishes:
|
||
an origin, a recording store, a relay with a DVR window, a cache, an edge. Every store holds
|
||
every covered subject until it purges it (§7.1). A policy whose `clocks` is not the link hash
|
||
of the registry the verifier is given is `registry_mismatch`; one whose `use` names a clock the
|
||
registry does not hold is `clock_unknown`.
|
||
|
||
### 5.2 The notice
|
||
|
||
A **notice**, `kind` `"wai-takedown-notice"`, signed by whoever submits it with a key of its
|
||
own:
|
||
|
||
| member | type | meaning |
|
||
|---|---|---|
|
||
| `kind` | string | `"wai-takedown-notice"` |
|
||
| `type` | string | `request`: a person's request for removal, or a notice of illegal content; `order`: an order of an authority or a court |
|
||
| `targets` | array | 1 to 16 subjects (§3), ascending, none twice: the exact location of the information |
|
||
| `ground` | string | `illegal`: illegal content; `terms`: incompatible with the service's terms |
|
||
| `ground_ref` | text, 0 to 500 | the provision or clause the submitter relies on |
|
||
| `statement` | text, 1 to 5000 | the submitter's explanation of its reasons |
|
||
| `contact` | text, 0 to 500 | the submitter's name and electronic contact |
|
||
| `issued_at` | time | when the submitter signed it |
|
||
| `signer_pubkey`, `sig` | | §2 |
|
||
|
||
`contact`, `statement` and the submitter's key are personal data. No export carries them (§9).
|
||
|
||
### 5.3 The takedown
|
||
|
||
A **takedown**, `kind` `"wai-takedown"`, signed by the service:
|
||
|
||
| member | type | meaning |
|
||
|---|---|---|
|
||
| `kind` | string | `"wai-takedown"` |
|
||
| `policy` | 32 B hex | the link hash of the policy it is taken under |
|
||
| `notice` | 32 B hex or `null` | the link hash of the notice or order it answers; `null` for the service's own initiative |
|
||
| `clock` | clock identifier or `null` | the clock it runs under, one of the policy's `use`; `null` for none |
|
||
| `clock_start` | time or `null` | the instant the clock starts, by its `starts`: when the service received the notice or order, or when it communicated its statement of reasons; `null` exactly when `clock` is |
|
||
| `reply_at` | time or `null` | under a `reply_window` clock, the instant a reply came, or `null` for none; `null` under any other clock |
|
||
| `targets` | array | 1 to 16 subjects, ascending, none twice: the sources it takes down |
|
||
| `verdict` | string | `remove`: no holder keeps the covered bytes; `disable`: no holder serves them |
|
||
| `exact` | array | 1 to 4096 subjects, ascending, none twice: the derivation closure of `targets` as the service computed it (§6) |
|
||
| `matched` | array | 0 to 256 perceptual matches, in ascending order of `match`, none twice: each `{ "match", "candidate" }`, the link hash of a match record (§5.4) and the content hash of its candidate |
|
||
| `ground` | string | `illegal` or `terms`, as a notice's |
|
||
| `category` | integer | 1 to 16: the category of live-moderation §9.5 |
|
||
| `automated` | object | `{ "detection", "decision" }`: `detection` `yes` when automated means detected or identified the item, `no` otherwise; `decision` `not`, `partially` or `fully`, how far automated means took the decision |
|
||
| `explanation` | text, 1 to 2000 | why the item is illegal, or incompatible, on that ground |
|
||
| `facts` | text, 1 to 5000 | the facts and circumstances relied on |
|
||
| `decided_at` | time | when the service took the decision |
|
||
| `signer_pubkey`, `sig` | | §2 |
|
||
|
||
Its rules, against what the verifier is given:
|
||
|
||
1. Its `policy` is the link hash of the policy given (`policy_mismatch`), and it is signed by
|
||
that policy's key (`not_authority`).
|
||
2. A `notice` that is not `null` names a notice the verifier is given (`notice_unknown`); each
|
||
of the takedown's `targets` is one of the notice's, and the notice's `issued_at` is at or
|
||
before `decided_at` (`notice_mismatch`). A takedown may act on part of a notice.
|
||
3. A `clock` that is not `null` is one of the policy's `use` (`clock_unknown`). Then, otherwise
|
||
`clock_invalid`: `clock_start` is `null` exactly when `clock` is; `clock_start` is at or
|
||
before `decided_at`; under a clock that starts at `receipt`, `notice` is not `null` and the
|
||
notice's `issued_at` is at or before `clock_start`; `reply_at` is `null` unless the clock's
|
||
`type` is `reply_window`, and, when it is not `null`, at or after `clock_start`.
|
||
|
||
A takedown is not superseded in this revision; a service that reinstates a disabled object
|
||
does so outside these records.
|
||
|
||
### 5.4 The match
|
||
|
||
A **match**, `kind` `"wai-takedown-match"`, signed by the party that ran the matcher, states
|
||
that one object resembles another by a method that is not exact:
|
||
|
||
| member | type | meaning |
|
||
|---|---|---|
|
||
| `kind` | string | `"wai-takedown-match"` |
|
||
| `source` | 32 B hex | the content hash of the covered object it was matched against |
|
||
| `candidate` | 32 B hex | the content hash of the object found; not `source` |
|
||
| `method` | term | the matching method (jwp-receipts §2.4.3's term) |
|
||
| `distance`, `threshold` | integers | the distance the method measured, and the threshold under which it calls a match; `distance` at most `threshold` |
|
||
| `issued_at` | time | when it was signed |
|
||
| `signer_pubkey`, `sig` | | §2 |
|
||
|
||
### 5.5 The purge
|
||
|
||
A **purge**, `kind` `"wai-takedown-purge"`, signed by a holder:
|
||
|
||
| member | type | meaning |
|
||
|---|---|---|
|
||
| `kind` | string | `"wai-takedown-purge"` |
|
||
| `takedown` | 32 B hex | the link hash of the takedown it answers |
|
||
| `subjects` | array | 1 to 4096 subjects, ascending, none twice: what it purged |
|
||
| `purged_at` | time | the instant from which it holds none of their bytes (under `remove`) or serves none of them (under `disable`) |
|
||
| `signer_pubkey`, `sig` | | §2 |
|
||
|
||
A holder that purges in several steps signs several purges. A purge that names another
|
||
takedown, or whose `purged_at` is after the instant of the verification, is `purge_mismatch`.
|
||
|
||
### 5.6 The counts
|
||
|
||
A **counts** record, `kind` `"wai-takedown-counts"`, signed by the service, states its
|
||
takedowns of a period by clock (§9):
|
||
|
||
| member | type | meaning |
|
||
|---|---|---|
|
||
| `kind` | string | `"wai-takedown-counts"` |
|
||
| `policy` | 32 B hex | the link hash of the policy its takedowns are taken under |
|
||
| `from`, `to` | times | the period: takedowns whose `decided_at` is at or after `from` and before `to`; `from` before `to` |
|
||
| `as_of` | time | the instant at which each takedown's outcome is computed |
|
||
| `takedowns` | array | 0 to 4096 link hashes, ascending, none twice: the takedowns it counts |
|
||
| `rows` | array | the rows (§9.1), in their order, none twice |
|
||
| `issued_at` | time | when it was signed |
|
||
| `signer_pubkey`, `sig` | | §2 |
|
||
|
||
## 6. The derivation closure
|
||
|
||
### 6.1 Seeds and rules
|
||
|
||
The **closure** of a takedown is the least set of subjects that holds its `targets` and is
|
||
closed under these rules, over the derivation records, origin claims and delivery receipts the
|
||
verifier is given:
|
||
|
||
1. **A group's objects.** For a group in the closure, each object of the group (§6.3) is in
|
||
it, as an `object` subject.
|
||
2. **A derivation.** For a derivation record (deriving-edge §5) any of whose inputs'
|
||
`content_hash` is an object in the closure, its output's `content_hash` is in it. The record
|
||
is a **closure record**.
|
||
3. **A claim.** For an origin claim (synthesis §3.1) any of whose inputs' `content_hash` is an
|
||
object in the closure, its `subject` is in it: an object, a group (which rule 1 then
|
||
expands) or a track. The claim is a **closure claim**. A claim whose subject is `samples`
|
||
(synthesis §3.1: a picture a party presents and does not publish as an object) adds
|
||
nothing: such samples are no item a holder serves or keeps for others.
|
||
|
||
A verifier computes it by applying the rules until none adds a subject. The set is finite and
|
||
every rule only adds, so the computation ends, and its result does not depend on the order the
|
||
rules are applied in. The closure is the exact set of what the records the verifier is given
|
||
derive, directly or through each other, from the targets: every transcode, ladder rung,
|
||
rebuild, crop or thumbnail, scaling, upscale, re-package and generated object whose record or
|
||
claim names a covered input, and every derivative of those. Nothing upstream of a target is
|
||
covered: a takedown of a derivative leaves its source alone.
|
||
|
||
A record or claim reads under its own extension's rules (deriving-edge §5.3, synthesis §2 and
|
||
§3.1) and its signature verifies, otherwise it is `evidence_invalid`; it is taken as its
|
||
signer's statement of what it derived from what. This extension does not check a record's
|
||
delivery receipts, re-execute its derivation, or check a claim's class: those are the checks of
|
||
deriving-edge §7 and synthesis §11, which a verifier may run first.
|
||
|
||
### 6.2 What the closure does not reach
|
||
|
||
- **Copies outside WAI's derivations.** A copy made by a party that signed no derivation record
|
||
or origin claim (a re-encode by a tool outside WAI, a screen recording, a re-upload of
|
||
transformed bytes) has a content hash no record names. The closure cannot reach it, and this
|
||
extension states so: such a copy is out of reach of the exact closure. A copy of the very same
|
||
bytes is reached, wherever it is, since its content hash is the subject's (§3).
|
||
- **Through a track.** No record of this revision binds a track's digest to the receipts of
|
||
its groups, so a track in the closure is covered by its name, and the closure does not reach
|
||
through it to its objects or their derivatives. A takedown that must reach them names the
|
||
track's groups or objects as targets beside it.
|
||
- **Through a take.** A take's chunks are named by their leaves (capture-binding §4.2), which
|
||
no derivation record or origin claim names. A take is covered by its id; the objects its
|
||
chunks were delivered as, where a party published them under receipts, are subjects of
|
||
their own.
|
||
|
||
### 6.3 A group's objects
|
||
|
||
A group in the closure is expanded from its delivery receipts: the verifier holds a group
|
||
receipt whose link hash is the group's `receipt` (jwp-receipts §3 step 3), whose signature
|
||
verifies (jwp-receipts §3 step 2) and whose `leaf_count` is at most 4096, and, for each
|
||
position 0 to `leaf_count` − 1, an object receipt of that group (its `track_alias` and
|
||
`group_id`) at that `object_id`, included at its position under the group's root
|
||
(jwp-receipts §3 step 5). The group's objects are those receipts' content hashes. A group the
|
||
verifier cannot expand so adds no object to the closure, and, once the closure is computed, a
|
||
group in it that was not expanded is `group_unexpanded`, at the first such in ascending order
|
||
of `receipt`: without its objects the exact set cannot be computed.
|
||
|
||
### 6.4 The takedown's stated set
|
||
|
||
The takedown's `exact` **MUST** be the closure. A verifier compares them:
|
||
|
||
- a subject of `exact` that is not in the closure is `match_as_exact` when it is an object that
|
||
is the `candidate` of a match the verifier is given, and `closure_unsupported` otherwise: a
|
||
takedown that claims to reach what no record derives from its targets is refused, and so is
|
||
one that presents a perceptual match as exact;
|
||
- a subject of the closure that is not in `exact` is `closure_incomplete`: a derivative left out.
|
||
|
||
### 6.5 Perceptual matches
|
||
|
||
A perceptual match is not exact: a method that compares content can miss a copy and can match
|
||
an unrelated one. A service that removes a matched object as well lists it in `matched`, never
|
||
in `exact`, beside the match record that found it. Each entry of `matched` names a match the
|
||
verifier is given whose `candidate` is the entry's (`match_unknown`), and whose `source` is an
|
||
object of the closure (`match_unanchored`). The matched class is purged and stated apart from
|
||
the closure (§7.4), and no clock's outcome rests on it.
|
||
|
||
## 7. Holders and purges
|
||
|
||
### 7.1 Who must purge
|
||
|
||
A holder must purge a covered subject when the verifier holds evidence that it held it. The
|
||
pairs of a holder and a subject it must purge are the **required** pairs:
|
||
|
||
| role | the holder | the subjects |
|
||
|---|---|---|
|
||
| `store` | each of the policy's `stores` | every subject of the closure |
|
||
| `deriver` | the signer of each closure record | the record's inputs whose content hash is in the closure, and its output |
|
||
| `claimant` | the signer of each closure claim | the claim's inputs whose content hash is in the closure, and its subject |
|
||
| `publisher` | the signer of each group receipt the closure's groups are expanded from (§6.3) | that group |
|
||
| `served` | the `upstream` of each acknowledgement or declaration (settlement §3, §4) whose item lists name an object of the closure | each such object |
|
||
| `received` | the `downstream` of each acknowledgement whose `role` is a hop (settlement §3.1) and whose item lists name an object of the closure | each such object |
|
||
|
||
A party that served a covered object (a cache, a relay or an edge that delivered it to a next
|
||
hop or a sink) held it, and the party downstream of it says so when it acknowledges what it
|
||
received: the acknowledgement is signed by the receiver, so a party cannot make itself or
|
||
another a holder by its own statement, except in a declaration of its own delivery, which makes
|
||
only itself one. A hop that acknowledges what it received held it too; a sink's copy is the
|
||
viewer's (§13) and no sink is held to purge. A party that served a covered object and that no
|
||
acknowledgement names is held to purge it only as a store of the policy.
|
||
|
||
### 7.2 What a purge covers
|
||
|
||
A purge by key *K* **covers** a required pair (*K*, *s*) when *s* is among its `subjects`; when
|
||
*s* is an object and its `subjects` name a group of the closure that holds *s* (§6.3); or when
|
||
*s* is a group and its `subjects` name every object of that group. A pair's **purge time** is
|
||
the earliest `purged_at` of the purges that cover it. A purge's subject that is in no required
|
||
pair of its signer, in the closure's class or the matched class, is **uncounted**: a purge by a
|
||
party that never held what it names removes nothing anyone was held to, and it counts for
|
||
nothing.
|
||
|
||
The removal is **complete** when every required pair has a purge time. The **first** and
|
||
**last purge** are the least and the greatest purge times of the required pairs.
|
||
|
||
### 7.3 The time to removal and the clock
|
||
|
||
A takedown's clock **starts** at its `clock_start`, or, with no clock, at its `decided_at`. Its
|
||
**time to removal**, for a complete removal, is the last purge less the start, in seconds, or 0
|
||
when the last purge is before the start; for one that is not complete, it has none. The
|
||
verification is made at an instant, `as_of`, given with the case (§8.1). By the clock's `type`
|
||
in the registry:
|
||
|
||
- **deadline**: the **deadline** is the start plus `duration_s`. A complete removal whose last
|
||
purge is at or before the deadline has **met** its clock; one whose last purge is after it is
|
||
`deadline_missed`. A removal that is not complete is `purge_missing` when `as_of` is after the
|
||
deadline, and **pending** otherwise.
|
||
- **reply_window**: the window **ends** at the start plus `duration_s`, or at `reply_at` when
|
||
that is earlier. A required pair whose purge time is before the window's end is
|
||
`window_not_observed`. Otherwise a complete removal has **met** its clock, and one that is not
|
||
complete is **pending**.
|
||
- **unfixed**, or no clock: a complete removal is **unfixed**: it has a time to removal and no
|
||
clock to hold it to; one that is not complete is **pending**.
|
||
|
||
The outcome is a computation over signed instants. It does not say that a regime applied, that
|
||
its duration was met in law, or that an item was unlawful; it says when each holder stated it
|
||
stopped holding what, against the clock the service named.
|
||
|
||
### 7.4 The matched class
|
||
|
||
The matched class's required pairs are each of the policy's stores with each `candidate` of
|
||
`matched`, and each party an acknowledgement shows served or, as a hop, received a candidate
|
||
(§7.1's `served` and `received` rules) with that candidate, covered by a purge that names the
|
||
candidate. Its missing pairs and last purge are
|
||
stated apart (§8.3), and no outcome of §7.3 rests on them.
|
||
|
||
## 8. Verification
|
||
|
||
### 8.1 Inputs
|
||
|
||
A takedown verifier is given a **case**:
|
||
|
||
- `registry`: the registry text (§4);
|
||
- `policy`: the policy text;
|
||
- `notices`: notice texts, in a list;
|
||
- `takedown`: the takedown text;
|
||
- `records`: derivation record texts (deriving-edge §5), in a list;
|
||
- `claims`: origin claim texts (synthesis §3.1), in a list;
|
||
- `acknowledgements`: acknowledgements and declarations (settlement §3, §4), each its record's
|
||
bytes and the item list of each of its entries, in order;
|
||
- `receipts`: group and object receipts (jwp-receipts §2.4.1, §2.4.2), as JSON values;
|
||
- `matches`: match texts, in a list;
|
||
- `purges`: purge texts, in a list;
|
||
- `as_of`: a time, the instant of the verification.
|
||
|
||
### 8.2 Steps
|
||
|
||
It runs these steps and stops at the first refusal, which it reports with the reason and its
|
||
place, a list of names and positions:
|
||
|
||
1. **The registry** reads (§4): at `["registry"]`.
|
||
2. **The policy** reads, its signature verifies, and §5.1's checks hold: at `["policy"]`.
|
||
3. **The notices**, in the list's order, each read and its signature verified: at
|
||
`["notice", i]`.
|
||
4. **The takedown** reads, its signature verifies, and §5.3's rules hold, in their order: at
|
||
`["takedown"]`.
|
||
5. **The records**, then **the claims**, then **the acknowledgements**, in their lists'
|
||
order (§6.1, `evidence_invalid`): at `["record", i]`, `["claim", i]`, `["acknowledgement", i]`.
|
||
An acknowledgement reads under settlement §3.6, its signature verifies, it is given one item
|
||
list for each of its entries, and each list reads and recomputes against its entry (settlement
|
||
§3.6); otherwise `evidence_invalid`.
|
||
6. **The matches**, each read and its signature verified: at `["match", i]`.
|
||
7. **The purges**, each read and its signature verified, and §5.5's checks: at `["purge", i]`.
|
||
8. **The closure** (§6.1, §6.3): a group that is not expanded, at `["group", g]` with *g* its
|
||
`receipt`.
|
||
9. **The stated set** (§6.4): each subject of `exact` in order, at `["exact", k]`; then each
|
||
subject of the closure in its order, at `["closure", k]`; then each entry of `matched`
|
||
(§6.5), at `["matched", k]`.
|
||
10. **The holders, the purges and the clock** (§7): the outcome.
|
||
|
||
### 8.3 Outcomes and what a verification states
|
||
|
||
- **refuse**: the reason and its place (§8.2).
|
||
- **accept**: the removal is complete and has met its clock, or has no fixed one.
|
||
- **pending**: the removal is not complete, and no deadline has passed.
|
||
- **missed**: `purge_missing`, `deadline_missed` or `window_not_observed` (§7.3), the first of
|
||
them that holds in that order.
|
||
|
||
With accept, pending or missed, the verifier states:
|
||
|
||
- `closure`: the closure, ascending;
|
||
- `holders`: each holder of a required pair, in ascending order of `key`, with its `roles`
|
||
(ascending, among `claimant`, `deriver`, `publisher`, `received`, `served` and `store`) and
|
||
the number of its `required` pairs;
|
||
- `missing`: each required pair with no purge time, as `{ "holder", "subject" }`, in ascending
|
||
order of the holder's key, then of the subject;
|
||
- `uncounted`: each uncounted subject, as `{ "purge", "subject" }`, the purge's position in
|
||
the list and the subject, in that order;
|
||
- `clock`: its identifier or `null`; `start`: the start's value; `deadline`: the deadline's
|
||
value, or `null` for a clock of another type; `window_end`: the window's end, or `null`;
|
||
- `first_purge`, `last_purge`: values, or `null` with no purge time;
|
||
- `time_to_removal_s`: the time to removal, or `null`;
|
||
- `verdict`: `met`, `missed`, `pending` or `unfixed`;
|
||
- `matched`: `{ "candidates", "missing", "last_purge" }`: the candidates (ascending), the
|
||
matched class's missing pairs and its last purge.
|
||
|
||
A value of a time is its number of seconds (§2).
|
||
|
||
## 9. Transparency counts
|
||
|
||
### 9.1 The rows
|
||
|
||
A counts record states one **row** per combination of clock, ground, category and source among
|
||
the takedowns it counts:
|
||
|
||
| member | type | meaning |
|
||
|---|---|---|
|
||
| `clock` | clock identifier or `null` | the takedowns' clock |
|
||
| `ground` | string | `illegal` or `terms` |
|
||
| `category` | integer | the takedowns' category (live-moderation §9.5) |
|
||
| `source` | string | `order` (a notice of type `order`), `request` (a notice of type `request`) or `own` (no notice) |
|
||
| `takedowns` | integer | the takedowns of the row, at least 1 |
|
||
| `met`, `missed`, `pending`, `unfixed` | integers | how many of them have each verdict at `as_of`; their sum is `takedowns` |
|
||
| `automated` | integer | how many of them were decided partially or fully by automated means |
|
||
| `median_s` | integer or `null` | the lower median of the times to removal of the row's takedowns that have one: of *n* values in ascending order, the one at position ⌊(*n* − 1) / 2⌋ from 0; `null` when none has one |
|
||
|
||
The rows are in ascending order of `clock` (`null` first, then identifiers in ascending byte
|
||
order), then `ground`, then `category`, then `source`, as strings or integers.
|
||
|
||
### 9.2 Recomputation
|
||
|
||
A counts verifier is given the counts text and the cases of its takedowns. It verifies each
|
||
case by §8 at the counts' `as_of`, in place of the case's own; a case refused is the counts'
|
||
refusal, at `["case", i]` followed by the case's place. Each case's policy is the counts'
|
||
`policy` (`policy_mismatch`) and the counts are signed by that policy's key (`not_authority`),
|
||
at `["counts"]`. The cases' takedowns are exactly those of `takedowns`, each `decided_at` in the
|
||
period, and the rows recompute from the cases' outcomes, member for member: otherwise
|
||
`counts_mismatch`, at `["counts"]`.
|
||
|
||
### 9.3 With the statement-of-reasons records and the transparency reports
|
||
|
||
The counts are compatible with live-moderation §9's statement-of-reasons export where the two
|
||
meet: a takedown's `ground`, `category` and `automated` carry the values that export maps to
|
||
`decision_ground`, `category`, `automated_detection` and `automated_decision`, and a service that
|
||
submits a statement of reasons for a takedown gives it the takedown's link hash, in 64
|
||
lowercase hexadecimal digits, as its `puid`, so that each row's takedowns can be found among
|
||
the records submitted. The rows split takedowns by source as Regulation (EU) 2022/2065, Article
|
||
15(1)(a) to (c), splits orders, notices and the provider's own initiative, by ground as point
|
||
(b) splits law from terms and conditions, and by category; `median_s` is the median time the
|
||
same points ask for, computed over what the purges show. This revision writes no
|
||
statement-of-reasons record for a takedown: the database's `content_type`, `territorial_scope`
|
||
and `content_date` have no source in these records.
|
||
|
||
## 10. Stores and relays
|
||
|
||
### 10.1 The reference relay's DVR store
|
||
|
||
The reference relay (`joule-moq-relay`, in `joule-code`) keeps a DVR window of a room's tracks
|
||
on disk (live-recording §8). It honours a signed takedown as it forgets a room whose objects
|
||
are sealed:
|
||
|
||
- **Where it reads takedowns.** A takedown rides as an object on the track whose namespace is
|
||
two fields, a key in 64 lowercase hexadecimal digits and `takedown`, and whose name is `0`.
|
||
The relay honours one whose `kind` is `wai-takedown`, whose `signer_pubkey` is the
|
||
namespace's key and one it is configured to trust (`dvr.takedown_keys`, and the keys of
|
||
`auth.authorities`), and whose signature verifies (§2). It reads `exact` and `matched` and
|
||
nothing else of it: it does not compute the closure, which is the verifier's.
|
||
- **What it forgets.** Every object whose content hash is an `object` of `exact` or a
|
||
`candidate` of `matched`, and every object of a `track` of `exact` from its `from_group` on,
|
||
is erased from the store under either verdict, its group written again with the object
|
||
among its erased; it is never stored again, and never served to a FETCH, from the store or
|
||
from the cache, to any session, a moderator's included. A group that becomes whole later
|
||
with a covered object stores it erased.
|
||
- **Across a restart.** The honoured takedowns are kept in `takedowns.wdl` and verified again on
|
||
start, and an erasure a crash left undone is made then.
|
||
- **Its purge.** Once the erasure is on disk the relay signs a purge (§5.5) with its key (the
|
||
key its session summaries and per-hop records are signed with), naming each `object` and
|
||
`track` of `exact` and each candidate of `matched`, at the instant the erasure was synced,
|
||
and writes it to `purges/` in the store's directory, under the takedown's link hash. A relay
|
||
named among a policy's stores answers its required pairs with it: a group of the closure is
|
||
covered by its objects (§7.2); a take is not, since the relay does not hold chunks by take.
|
||
- **What it does not do.** It does not check the live fan-out: an object published again after
|
||
a takedown, live, is forwarded to the subscribers of its track, and kept out of the store and
|
||
out of every FETCH.
|
||
|
||
### 10.2 Recording packages
|
||
|
||
A recording package ([live-recording](live-recording.md) §3) is a set of files written once
|
||
when it is sealed. A package sealed after a takedown leaves each covered chunk's file and each
|
||
covered event record out, as live-moderation §8.2 has a redaction left out; the manifest still
|
||
lists the chunk's descriptor. The reference does not rewrite a package already sealed: its
|
||
operator deletes the covered files and signs the purge. This revision implements no package
|
||
rewriter.
|
||
|
||
### 10.3 HTTP origins and caches
|
||
|
||
An object served over HTTP ([http-delivery](http-delivery.md)) is immutable and cacheable for
|
||
a year, so a cache that held one keeps it until it is purged by name. Each such cache that held
|
||
a covered object is a holder, and a takedown is complete only with its purge. The reference
|
||
`wai http origin` command serves the files of a directory it is given and keeps no store of its
|
||
own; its operator purges by deleting the files and signs the purge.
|
||
|
||
## 11. Reasons
|
||
|
||
The codes this extension adds, each with where it is raised:
|
||
|
||
| reason | raised at |
|
||
|---|---|
|
||
| `record_malformed` | §2, §8.2 |
|
||
| `bad_key` | §2, §8.2 |
|
||
| `bad_signature` | §2, §8.2 |
|
||
| `registry_mismatch` | §5.1, §8.2 step 2 |
|
||
| `clock_unknown` | §5.1, §5.3 rule 3 |
|
||
| `policy_mismatch` | §5.3 rule 1, §9.2 |
|
||
| `not_authority` | §5.3 rule 1, §9.2 |
|
||
| `notice_unknown` | §5.3 rule 2 |
|
||
| `notice_mismatch` | §5.3 rule 2 |
|
||
| `clock_invalid` | §5.3 rule 3 |
|
||
| `evidence_invalid` | §6.1, §8.2 step 5 |
|
||
| `purge_mismatch` | §5.5, §8.2 step 7 |
|
||
| `group_unexpanded` | §6.3, §8.2 step 8 |
|
||
| `closure_unsupported` | §6.4 |
|
||
| `match_as_exact` | §6.4 |
|
||
| `closure_incomplete` | §6.4 |
|
||
| `match_unknown` | §6.5 |
|
||
| `match_unanchored` | §6.5 |
|
||
| `purge_missing` | §7.3, an outcome of missed |
|
||
| `deadline_missed` | §7.3, an outcome of missed |
|
||
| `window_not_observed` | §7.3, an outcome of missed |
|
||
| `counts_mismatch` | §9.2 |
|
||
|
||
## 12. Conformance
|
||
|
||
[`live-conformance/takedown/`](../live-conformance/takedown/) holds the corpus, written from
|
||
fixed seeds by `wai_takedown_vectors gen`, in one file, `vectors.json`, with the registry of
|
||
§4 as the repository publishes it. Its cases share one service, one policy and one chain of
|
||
derivations from a source picture: a component exploded from it, a crop of that component as a
|
||
thumbnail and a scaling of it as a ladder rung (derivation records); a transcode of the
|
||
source, an upscale of the transcode, a generated group from the upscale and a live track from
|
||
the rung (origin claims); derivations of an unrelated object beside them; and two
|
||
acknowledgements: a viewer's of the thumbnail it received from a cache outside the policy's
|
||
stores, and that cache's, as a hop, of the component and the rung it received from the relay.
|
||
It has:
|
||
|
||
- **Removals**: a request under a 48-hour deadline met, with a perceptual match purged beside
|
||
the closure; an order under a 3-hour deadline missed by one holder's late purge, and the same
|
||
order still pending before its deadline; a removal past its deadline with a holder's purge
|
||
missing, and one with the purge of the cache the acknowledgements name missing; a removal of
|
||
the service's own initiative inside and after a 24-hour reply window,
|
||
and one whose window a reply ended early; a removal under the unfixed clock; a group as the
|
||
target, expanded from its receipts; and a take as the target.
|
||
- **Negative controls**, each refused or missed with its reason: a derivative left out of the
|
||
stated set (`closure_incomplete`); a purge after the deadline (`deadline_missed`, never
|
||
accepted); a purge signed by a party that never held the object, standing in for a holder's
|
||
missing one (`purge_missing`, the stranger's subjects uncounted); a perceptual match
|
||
presented as exact (`match_as_exact`); a takedown of an object no derivation names that
|
||
claims to have reached derivatives (`closure_unsupported`); and one case or more for each
|
||
other reason of §11, each a control of one defect.
|
||
- **Counts** over the removals, and one whose row does not recompute (`counts_mismatch`).
|
||
|
||
A conforming verifier gives every case the outcome, the reason, the place and the facts the
|
||
corpus states. `wai_takedown_vectors verify` and the test `tests/takedown.rs` check that the
|
||
file regenerates byte for byte, that `wai-rs` gives every case its outcome, that every reason
|
||
of §11 appears, and that §11 lists exactly the reference's codes.
|
||
`live-conformance/takedown_verify.py` is the second implementation, written from this text: it
|
||
shares no code with `wai-rs`; it reads derivation records and delivery receipts with the
|
||
deriving-edge corpus's second implementation (`edge-conformance/edge_verify.py`), origin claims
|
||
with the synthesis corpus's (`live-conformance/synthesis_verify.py`), acknowledgements with the
|
||
settlement corpus's (`live-conformance/settlement_verify.py`), and every signature with
|
||
WAI's strict rule (`claim-conformance/_ed25519.py`); it requires every mutated copy of the
|
||
corpus to fail, and, with each of its own checks dropped in turn, requires the corpus to catch
|
||
the weakened verifier. The relay's behaviour is tested by `tests/dvr.rs` in `joule-moq-relay`.
|
||
|
||
## 13. Security and privacy considerations
|
||
|
||
- **What a closure rests on.** A derivation record or an origin claim is its signer's
|
||
statement. A party that signs a false one can name another's object as the output of a
|
||
derivation from a target, and bring it into a closure. The verifier states which records
|
||
and claims made the closure and who signed them (§8.3, the holders' roles), and a verifier
|
||
that must not rely on a signer's word runs deriving-edge §7 over the records first, which
|
||
re-executes what is pinned and ties each output to its edge's receipts.
|
||
- **Who is held to purge.** An acknowledgement makes its upstream a holder of what it names, and
|
||
is signed by the receiver. A receiver that acknowledges an object it never received holds its
|
||
upstream to a purge of it, which that upstream answers with a purge of an object it does not
|
||
hold; the verifier states each holder's roles, and the acknowledgement is the receiver's
|
||
signed statement, as settlement §13 has it.
|
||
- **What a purge proves.** A purge is its holder's signed statement of when it stopped holding
|
||
or serving the subjects. Nothing in a record proves that bytes were erased from a disk; a
|
||
holder that signs a purge and keeps serving is caught by a party that fetches from it and
|
||
holds the purge. The time of a purge, of a notice and of a takedown are their signers'.
|
||
- **The clock's start.** `clock_start` is the service's statement of when it received the
|
||
notice or communicated its statement. It cannot be before the notice was signed (§5.3 rule
|
||
3). A submitter that holds a confirmation of receipt (Regulation (EU) 2022/2065, Article
|
||
16(4)) signed earlier holds the service to that instant.
|
||
- **Out of reach.** The exact closure reaches every copy of the same bytes and every derivative a
|
||
WAI record names; it does not reach a copy made outside them (§6.2). A perceptual match
|
||
reaches some of those and can match an unrelated object, so it is kept apart and never
|
||
counted as exact.
|
||
- **Withholding is not erasure everywhere.** A takedown stops what the holders that purge hold
|
||
and serve. A copy delivered to a viewer before the takedown stays with the viewer.
|
||
- **Personal data.** A notice holds its submitter's contact and statement. A verifier is given
|
||
notices to check takedowns against them, and no counts row, purge or export carries a
|
||
notice's texts or its key. A takedown names its targets by digest and holds no content; its
|
||
explanation and facts are published with it, and whoever writes them writes them for a
|
||
public record.
|
||
- **Keys.** A relay honours takedowns only from keys it is configured to trust; a takedown
|
||
from any other key changes nothing it holds.
|