open-standards/wai/extensions/exact-upscale.md

WAI Extension: Exact Upscale

Status: Draft. Registers the record wai.media.upscale: super-resolution as an exact derivation, and an account of which pixels a viewer saw were sent and which the sink made. A sink that receives a picture at a low resolution and presents it at a higher one runs wai-upscale:sr2 (or wai-upscale:sr4): the pinned upscale of integer-operators §5E E11 plus a detail an integer network computes from the picture, every operation one of wai-ops/1, under a prior the sink holds by its SHA-256 (SPEC §3.1). Every sink computes the same bytes, so the high-resolution picture is a function of the transmitted bytes and the prior, and anyone holding both recomputes it. Since the sink's upscale is known exactly, an origin can send a residual against it, a WIR1 payload of staged-delivery Appendix A, and sign an upscale record stating the picture it aimed at and how far the sink's output may lie from it: no sample at all, or a stated bound. A prior overfitted to one title, a per-title prior, is carried by its digest like any other. A sink's presentation statement says, for a span of what it presented, which pictures were transmitted, which it derived exactly, which it edited by a step no one can recompute and which it generated, with the pixels and frames of each; each sink-made picture carries an origin claim of synthesis, and a generated picture names the model that made it. No new transport. The reference implementation is the int_sr module of wai-rs (feature neural_int), for the derivations, and its exact_upscale module (feature exact_upscale), for the records; the corpus is live-conformance/upscale/, and its second implementation is live-conformance/upscale_verify.py. The tool that trains priors is tools/wai_int_sr_train.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

Texts read for this revision: the WAI texts this one cites, and RFC 8032 and RFC 8785.

1. Scope

In scope: the prior and the two derivations, their arithmetic, colour, edges and limits, and their graph in the integer operator set (§3); the residual and the upscale record (§4); per-title priors (§5); the categories of presented pictures, their origin claims and the presentation statement (§6); verification (§7).

Out of scope: a transport; how a prior is trained, which is not part of the derivation (§5); how a sink schedules a derivation; generating a picture between two others, whose arithmetic is the generator's and whose claim is synthesis's (§6.1); and the presentation of anything that is not a picture.

Relationship to the core spec: wai.media.upscale is a record (SPEC §5): a signed record a WAI tool emits and verifies. It names no payload format and is not dispatchable. A prior is a parameter set of one file, pinned by SHA-256 as SPEC §3.1 pins any other (§3.5).

Relationship to synthesis: wai-upscale:sr2 and wai-upscale:sr4 are pinned integer derivations of synthesis §4.3, and an assistive claim over one is proved by recomputation from the input's decoded samples and the prior (synthesis §4.4). A picture a sink presents and does not publish is named in a claim by the digest of its samples (synthesis §3.1).

Relationship to determinism-tiers: the derivations are integer throughout, so their output is a pure function of the input's samples and the prior's bytes, and a sink that presents it holds the tier of the decode it ran on (determinism-tiers §2, What a tier covers): decode-equivalence over an input at decode-equivalence.

2. Records

The records of this extension are JSON objects under the encodings of synthesis §2: byte strings in lowercase hexadecimal, unsigned integers of at most 2^53 − 1 in plain decimal, no other number, no boolean, terms and date-times as jwp-receipts §2.4.3 defines them, no member named twice, at most 32 levels of nesting, exactly the members a section lists at every level, and a kind naming the record; one that does not is record_malformed.

Every record carries signer_pubkey (32 bytes) and sig (64 bytes):

sig       = Ed25519(signer_key, "wai:upscale\x01" || JCS(record without "sig"))
link_hash = BLAKE3("wai:upscale-link\x01" || JCS(record))

The signature is verified strictly (SPEC §7 Signatures): a key that fails step 1 or 2 is bad_key, and a signature that fails a later step bad_signature. A record is named by its link hash. A picture's samples are its RGB8 buffer (rows top to bottom, pixels left to right, three bytes a pixel, no padding), and its digest is SHA-256 (FIPS 180-4) over them, as synthesis §2 has it. A picture of w × h has w · h pixels.

3. The derivations

3.1 The prior: WSR1

A prior is one file. Every multi-byte field is little-endian.

off	size	field	rule
0	4	magic WSR1
4	1	version	1
5	1	n_layers	1 to 8
6	2	act_hi	1 to 32767: the ceiling of every hidden activation
8	…	the layers	n_layers layers, one after another; nothing follows the last

A layer:

size	field	rule
1	k	1, 3 or 5: the kernel's side
1	shift	0 to 31
2	cout	1 to 64; the last layer's is 12
cout · cin · k · k	weights	signed bytes, in the order [cout][cin][k][k]
4 · cout	biases	signed 32-bit integers
2 · cout	multipliers	unsigned 16-bit integers, each at least 1

cin is 3 for the first layer and the previous layer's cout for every other.

A reader checks, in this order, and the first that fails names the refusal: the magic (bad_magic); the version, present and 1 (version); n_layers and act_hi (field_range); then each layer in turn: its four bytes present (truncated), its k, shift and cout (field_range), its arrays present (truncated), each multiplier at least 1 (field_range); then nothing after the last layer (trailing). A field missing from the 8-byte header is truncated. A file that fails any of these is not a prior; a record that rests on it is refused with the record's own reason (§7.2).

3.2 wai-upscale:sr2

wai-upscale:sr2 under a prior P maps a w × h picture in to a 2w × 2h picture out. All values are integers; no formula wraps or saturates unless it says so.

1. The base. base is up2_predict (integer-operators §5E E11) of in, channel by channel, at 2h × 2w: the arithmetic of wai-synthesis:upscale2 (synthesis §4.1), the last row and column repeated.

2. Centring. x₀[c][y][x] = in(x, y)[c] − 128, for c the red, green and blue bytes in that order: a tensor of 3 channels, h rows and w columns.

3. The layers. For each layer i of P, in order, with input xᵢ of cin channels:

   a[o][y][x] = Σ W[o][ci][ky][kx] · x̃ᵢ[ci][y + ky − ⌊k/2⌋][x + kx − ⌊k/2⌋],

   the sum over ci < cin, ky < k, kx < k, with x̃ᵢ equal to xᵢ inside the picture and 0 outside it; then v = R_half_up((a + bias[o]) · mult[o], shift) (integer-operators §2). For every layer but the last, xᵢ₊₁ = clamp(v, 0, act_hi). The last layer's v, of 12 channels, is the detail d.

4. Sub-positions. detail[c][2y + i][2x + j] = d[4c + 2i + j][y][x], for i, j ∈ {0, 1}: pixel_shuffle with factor 2 (integer-operators §5.10).

5. The output. out(x, y)[c] = clamp(base(x, y)[c] + detail[c][y][x], 0, 255).

A prior whose last layer's weights and biases are all 0 gives out = base.

3.3 Colour, edges, scale and limits

• Colour. The derivation reads and writes RGB8 samples, the three channels together: each layer mixes all of its input's channels. It states no colour space and converts none: its output is in its input's.
• Edges. The base repeats the last row and column (§3.2 step 1). The network reads zero outside the picture after centring (step 3), which is the sample value 128 before it.
• Scale. wai-upscale:sr2 doubles each side. wai-upscale:sr4 under P is wai-upscale:sr2 under P of wai-upscale:sr2 under P: a 4w × 4h output. No other factor is registered.
• Limits. A derivation is defined for a picture of at least one pixel whose every tensor, in either pass of sr4, holds at most 2^26 elements (integer-operators §3): for sr2, w · h times the largest cout of P, and 12 · w · h, are at most 2^26. A picture past that has no output under P.

3.4 Through the integer operator set

The derivation is one WHG1 graph (integer-operators §10) and one stage operator. The lowering sr2/1 writes, for a prior P:

• two data inputs, sizes set per decode: tensor 0, the picture [3, h, w], and tensor 1, its base [3, 2h, 2w], each with the policy check over [0, 255];
• then the operators, each one's output taking the next tensor index, every tensor derived with no exponent: rescale of tensor 0 with no multiplier, shift 0 and a pre bias of −128 on each channel; for each layer, conv with its cout, kernel k × k, stride 1, zero padding ⌊k/2⌋ on each side, one group, its weights at one byte, and an rq with one multiplier per channel at four bytes, its one shift, a pre bias per channel at four bytes and the rounding half_up; after each layer but the last, clamp(0, act_hi); then pixel_shuffle(2); add of tensor 1 and the detail; clamp(0, 255), the graph's one output.

The driver computes the base with up2_predict (§5E E11), supplies both inputs, and reads the output back as RGB8. A sink loads the graph from its canonical bytes under acc64 with its bounds certificate (integer-operators §8): a prior that reads (§3.1) always meets acc64, since no value it forms passes 2^63. Where the certificate also meets acc32, a fabric of wai-ops/1-i32 (integer-operators §9.1) runs it, and every operator of the graph is in that subset. The outputs are the same integers on every fabric (integer-operators §4).

3.5 Pinning a prior

A prior is a parameter set of one file, sr.wsr, and SPEC §3.1 pins it: sha256 is the SHA-256 of the file's bytes, with no form. Its determinism is int-sr2 and its tier, where stated, decode-equivalence. Nothing but the digest names a prior: not an id, not a title, not a network that compares equal after reading. A sink that runs a derivation under a prior it does not hold by that digest has run another derivation.

4. The residual and the upscale record

4.1 The residual

A residual is a WIR1 payload (staged-delivery Appendix A) of the RGB8 unit kind with predictor 0 (identity), n_units 1, d0 and d1 the output's height and width, and d2 3. Its decode is Appendix A.4's, with the upscaled picture as the unit's current output: each sample becomes clamp(u + r · q, 0, 255), u the upscaled sample and r the decoded residual. A residual whose every r is the target's sample less u, divided by q and rounded to the nearest integer, halves away from zero (staged-delivery A.4's rdiv), brings every sample within ⌊q/2⌋ of the target's, and to it exactly when q is 1.

4.2 The upscale record

An upscale record, kind "wai-upscale", signed by the party that sends the picture (the origin), says what a sink presents for one picture:

member	type	meaning
kind	string	"wai-upscale"
object	32 B hex	the content hash (jwp-receipts §2.1) of the object whose decode is the input picture
input	object	{"width", "height", "samples"}: the input picture's size, each at least 1, and digest
derivation	string	wai-upscale:sr2 or wai-upscale:sr4
prior	32 B hex	the SHA-256 of the prior (§3.5)
upscaled	32 B hex	the digest of the derivation's output
residual	object or null	{"sha256", "q"}: the SHA-256 of the residual's bytes and its q, 1 to 65535; null for none
output	32 B hex	the digest of the picture a sink presents: upscaled with no residual; the residual's decode over the upscaled picture otherwise
target	object or null	with a residual, {"samples", "bound"}: the digest of the picture the origin aimed at, and the largest difference it states between any sample of the output and the target's, 0 to 255; null exactly when residual is
issued_at	date-time	when the signer signed it
signer_pubkey, sig		§2

A record whose output would pass 2^26 bytes (4 · w · h · 3 for sr2, 16 · w · h · 3 for sr4), whose residual and target are not both null or both objects, or with no residual whose output is not its upscaled, is record_malformed.

A record with a bound of 0 says the sink presents the target, byte for byte. A record with a residual and a larger bound says the sink presents a picture no sample of which lies further than the bound from the target's: the target is the origin's, and what reaches the viewer is within the bound of it.

5. Per-title priors

A per-title prior is a prior an origin made for one title: a set of frames, a programme, a game's art. It is a WSR1 file like any other (§3.1), pinned by its digest (§3.5), carried to a sink as any parameter set is (prior-carriage), and named by the records over the title's pictures. Nothing in a sink's derivation depends on how the prior was made.

How a prior is trained is not part of this extension. A prior trained in floating point and quantised to WSR1 is exact all the same: the derivation is the integer arithmetic of §3.2 over the prior's bytes. A tool that trains priors SHOULD record its seeds and inputs, and say whether its training regenerates the same bytes; where it does not, the prior's bytes, pinned by their digest, are what is reproducible. The reference tool, tools/wai_int_sr_train.py, trains in floating point from fixed seeds, and its training is not claimed to regenerate the same bytes on another machine.

An origin SHOULD send a per-title prior only where the bytes it saves over the title, at the quality the origin delivers (its residuals' q and the bounds it states), exceed the prior's own bytes plus the bytes of the records that name it. A prior that saves less costs every sink that fetches it more than it returns.

6. Accounting for sink-made pictures

6.1 Categories and their claims

Every picture a sink presents is in one category:

category	what the picture is	its origin claim (synthesis §3)
transmitted	the decode of an object the sink received, or that decode upscaled and refined by a residual to the target of an upscale record (§4.2)	the object's own claim; for a refined picture, a convert claim whose subject is the picture's samples and whose one input is the object the record names
derived	wai-upscale:sr2 or wai-upscale:sr4 of a transmitted picture, with no residual	an assist claim whose subject is the picture's samples, with replicate evidence naming the derivation and its prior
asserted	an edit of a transmitted picture by a step no verifier recomputes: an upscale or an enhancement in floating point	an assist claim whose subject is the picture's samples, with attest evidence
synthesized	a picture a model made: a frame generated between two transmitted frames, a frame that replaces a lost one	a generate claim whose subject is the picture's samples and whose models include the model that made it

The sink signs the claim of each picture it made, as the party that performed the step (synthesis §3.1); the origin signs the claim of a refined picture, whose samples its record fixes. A derived picture is proved by recomputation from the input's samples and the prior (synthesis §4.4); an asserted one is the sink's statement and nothing more; a synthesized one is never labelled lower than synthesized (synthesis §3.2).

6.2 The presentation statement

A sink signs a presentation statement, kind "wai-presentation", for a span of a viewing session, under a key of that session (no identity of the viewer or the device):

member	type	meaning
kind	string	"wai-presentation"
session	32 B hex	fresh random bytes of the session
from, to	integer	the span, in microseconds of the session's clock, from < to
frames	array	1 to 4096 entries, one for each picture presented in the span, in presentation order
totals	object	{"frames", "pixels"}, each a count: {"transmitted", "derived", "asserted", "synthesized"} of integers
issued_at	date-time	when it signed it
signer_pubkey, sig		§2

An entry:

member	type	meaning
at	integer	when it was presented: from ≤ at < to, and greater than the entry before it's
made	string	its category (§6.1)
width, height	integer	its size, each at least 1, their product at most 2^40
samples	32 B hex	its digest
object	32 B hex or null	the content hash of the object it was decoded from or computed from; null exactly for synthesized
claim	32 B hex	the link hash of its origin claim (synthesis §2)
upscale	32 B hex or null	the link hash of the upscale record it follows, where it follows one; null for asserted and synthesized
made_by	32 B hex or null	for derived, the prior's SHA-256; for asserted and synthesized, the digest of the model that made it, which for asserted may be null; null for transmitted. For synthesized, this is the slot made_by of the contribution record wai.media.contribution, which names the party, here the sink's model, that made a part of the media.
pixels	object	a count (above): how many of its pixels are in each category (§6.3)

A statement that breaks a rule of this table is record_malformed.

6.3 Counting pixels and frames

An entry's pixels are counted by its category, with W · H its own pixels and n the pixels of the picture it was computed from:

• transmitted: all W · H are transmitted;
• derived: n are transmitted and W · H − n derived;
• asserted: min(n, W · H) are transmitted and the rest asserted;
• synthesized: all W · H are synthesized.

n is the input size the claim's evidence states. So a picture upscaled twice on each side counts a quarter of its pixels as transmitted, whatever it shows: the transmitted bytes carried that many. A refined picture counts all of its pixels as transmitted: the origin fixed every one of them, within its stated bound.

The statement's totals.frames counts the entries of each category, and totals.pixels adds the entries' pixels. The fraction of the span's pixels, or frames, in a category is its count over the sum of the four. A statement whose totals are not these sums is fractions_mismatch.

7. Verification

7.1 Inputs

A verifier is given, each in a list: origin claims (synthesis §3.1); upscale records; presentation statements; the envelopes of the objects it holds, each named by its content hash; the decoded samples it holds, each with the content hash of the object decoded; the priors it holds, each named by its SHA-256; the residuals it holds, each named by its SHA-256; the targets it holds, each a picture named by its digest; and a policy of synthesis §11.1's members and one more:

member	values	default	meaning
upscale	accept-unrecomputed, require-proof	accept-unrecomputed	require-proof: §7.2 step 4.6 applies (upscale_not_proved)

7.2 Steps

The verifier runs these steps and stops at the first refusal, which it reports with the reason and where: a record by its list and index (["upscale", i], ["presentation", i]), an entry of a statement by both (["presentation", i, "frame", j]), or synthesis's own place in the claims (["synthesis", …]).

1. Reading. Each upscale record, then each presentation statement, reads (§2, §4.2, §6.2) and its signature verifies: record_malformed, bad_key, bad_signature.

2. Origin claims. Synthesis §11.2 steps 1 and 2 over the claims, with the objects, the decoded samples, the priors and the policy: a refusal there is this verification's, at ["synthesis", ["claim", i]].

3. Priors. Nothing yet: a prior is read where a record names it (step 4).

4. Upscale records, in order, each by these checks in order:

   1. Its prior, where the verifier holds one by prior: it reads (§3.1) (prior_invalid).
   2. Its input, where the verifier holds the decoded samples of object: they are width · height · 3 bytes and their digest is input.samples (input_samples_mismatch).
   3. With the prior and the input held: the derivation's output (§3.2, §3.3) has the digest upscaled (upscale_mismatch). An input past §3.3's limits under the prior has no output, and is refused here too.
   4. Its residual, where the verifier holds one by residual.sha256: it reads as §4.1's payload for the output's size with the record's q, and decodes to its end (staged-delivery A.3, A.4 steps 2 and 3) (residual_invalid); with the upscaled picture recomputed, its decode over that picture has the digest output (output_mismatch).
   5. Its target: with a bound of 0, output is target.samples; with the output recomputed and the target held, no sample of the output lies further from the target's than the bound (residual_short).
   6. Under upscale require-proof, a record whose output was not recomputed in steps 3 and 4 (with no residual, step 3; with one, steps 3 and 4): upscale_not_proved.

   A record whose output was recomputed is proved; otherwise it is not recomputed.

5. Presentation statements, in order, each entry in order, by these checks in order:

   1. Its claim is a claim the verifier holds (claim_unknown), and, where it names one, its upscale record is one it holds (upscale_unknown).
   2. The claim is about the entry's picture: an object claim of the entry's object, for a transmitted entry with no upscale record; a claim whose subject is the entry's samples, for every other (frame_subject_mismatch).
   3. The claim's class is not below the category's (§6.1): assistive for derived and asserted, synthesized for synthesized (made_understated). A sink's upscaled picture claimed captured or reconstructed, and a generated picture claimed assistive, are refused here.
   4. The claim is the category's (§6.1) (made_mismatch):
      • transmitted with an upscale record: the record has a residual, its object is the entry's, its output is the entry's samples, and the claim is a convert whose one input is that object;
      • derived: the claim is an assist whose evidence is replicate and names wai-upscale:sr2 or wai-upscale:sr4, whose one input is the entry's object, and whose output, the evidence's size scaled by the derivation's factor, is the entry's size; where the entry names an upscale record, the record's object and derivation are the evidence's and its upscaled is the entry's samples;
      • asserted: the claim is an assist whose evidence is attest, whose one input is the entry's object;
      • synthesized: the claim is a generate.
   5. For derived, made_by is the evidence's prior, and the upscale record's prior where the entry names one (prior_mismatch). For asserted and synthesized, a made_by that is not null is one of the claim's models, and for synthesized it is not null (made_by_mismatch).
   6. Where the verifier holds the decoded samples of a transmitted entry's object and the entry names no upscale record: their digest is the entry's samples (presented_samples_mismatch).
   7. Its pixels are §6.3's (pixels_mismatch).

   Then the statement's totals (§6.3, fractions_mismatch).

7.3 What an accepted case states

• the synthesis statements of the claims: for each, in order, its link hash, class and, for an assist claim, whether it is proved, not-recomputed or asserted (synthesis §11.3);
• for each upscale record, in order: its link hash, proved or not-recomputed, and, where the output was recomputed and the target held, the largest difference of any sample of the output from the target's, or null;
• for each presentation statement, in order: its link hash and its totals.

8. Reasons

The codes this extension adds, each with where it is raised:

reason	raised at
record_malformed	§2, §4.2, §6.2, §7.2 step 1
bad_key	§2
bad_signature	§2
prior_invalid	§3.1, §7.2 step 4.1
input_samples_mismatch	§7.2 step 4.2
upscale_mismatch	§7.2 step 4.3
residual_invalid	§4.1, §7.2 step 4.4
output_mismatch	§7.2 step 4.4
residual_short	§4.2, §7.2 step 4.5
upscale_not_proved	§7.2 step 4.6
claim_unknown	§7.2 step 5.1
upscale_unknown	§7.2 step 5.1
frame_subject_mismatch	§7.2 step 5.2
made_understated	§6.1, §7.2 step 5.3
made_mismatch	§6.1, §7.2 step 5.4
prior_mismatch	§3.5, §7.2 step 5.5
made_by_mismatch	§6.2, §7.2 step 5.5
presented_samples_mismatch	§7.2 step 5.6
pixels_mismatch	§6.3, §7.2 step 5.7
fractions_mismatch	§6.3

A verifier also raises every code synthesis raises over the claims (synthesis §12), at ["synthesis", …]: assistive_recompute_mismatch among them, for a sink's upscale whose claim states an output its derivation does not give.

9. Conformance

live-conformance/upscale/ holds the corpus: vectors.json, written from fixed seeds by wai_upscale_vectors gen, the priors it reads in priors/, each pinned by its SHA-256 in the generator, and sha256.json, the digest of every derivation's output in the corpus. It has:

• Accepted: sr2 of a picture under a prior trained on generated pictures, proved by recomputation in its upscale record and in the sink's assist claim; sr4 under a seeded prior; a residual that brings the output to its target exactly, and one to a bound of 2; sr2 under a per-title prior; and a presentation statement over a span that holds a transmitted picture, a derived one, an asserted float upscale and a frame generated between two others, with its totals.
• Refused: each reason of §8 at least once, each a control of one defect: among them an output that does not recompute (upscale_mismatch, and synthesis's assistive_recompute_mismatch), a prior other than the one the claim names (prior_mismatch), a residual that does not reach its target (residual_short), an upscaled picture claimed captured and a generated picture claimed assistive (made_understated), and a statement whose totals do not add up (fractions_mismatch).

A conforming verifier gives every vector the outcome, the reason, the place and the statements the corpus states. wai_upscale_vectors verify and the test tests/exact_upscale.rs check that the corpus regenerates byte for byte, that wai-rs gives every vector its outcome, and that §8 lists exactly the reference's codes. live-conformance/upscale_verify.py is the second implementation, written from this text: it shares no code with wai-rs; it runs the derivations from §3.2 in integer arithmetic of its own, verifies the claims with the synthesis corpus's second implementation (synthesis_verify.py), decodes residuals with the staged-delivery corpus's (staged_verify.py), reads §8's codes from this text, and mutates copies of the corpus and requires every copy to be refused.

10. Security and privacy considerations

• A derived picture is a function of the transmitted bytes and the prior, and nothing else: it can be checked, and it can be no more faithful than they are. What the prior adds is what it learned from the pictures it was trained on; a proved claim says the sink ran the derivation, not that the detail it added was in the scene.
• A prior's bytes are fetched by digest from whoever serves them. A sink that holds a prior under a digest has the bytes the origin pinned; it trusts no name, title or id.
• A residual's target is the origin's statement. A verifier that does not hold the target checks only a bound of 0, against the target's digest; the bound of a larger one rests on the origin's signature until a verifier holds the target.
• A presentation statement shows what the session's key reports. It is not evidence that a person watched, and it names no viewer, no device and no account. Its digests name pictures, which a party that holds the pictures can recognise.
• A sink can leave pictures out of its statement. A statement covers the entries it lists; a gap in at or a span with fewer frames than the content's rate is visible to a verifier that knows the rate.