3fac029a71
Each *-conformance/README.md now has a line that begins **Provenance.**: the generator that writes the corpus and its input, and for an input not generated here its source and licence. OER/2's corpus is settled and left as it is. byte-exact-conformance and quantum-ops-conformance get a README. wai-rs/tests/vectors/README.md gives a row per vector directory. Where the staged-delivery, staged-measurement and telemetry READMEs called their content "synthetic", that sentence is now their provenance line. Two vectors are recorded as not generated, from the provenance audit: int_video's video_real (a pan over a third-party picture) and neural-conformance's photo (a third-party picture, with a decoder trained on unrecorded images). Their lines say so and that they are to be replaced; the photo bullet no longer names the picture's source product. tests/corpus_provenance.rs requires the line in every corpus and a row for every vector directory; CI runs it with the carriage tests. CONTRIBUTING states the rule: corpus content is generated here, or recorded with a licence that permits redistribution, and a word like "synthetic" is not a provenance line. Co-Authored-By: Claude Opus 5.5 <[email protected]>
70 lines
4 KiB
Markdown
70 lines
4 KiB
Markdown
# Neural Conformance Corpus — integer synthesis
|
||
|
||
Vectors for the **integer-synthesis** neural decoder (`wai.neural.int_synth`).
|
||
This is the one neural decoder that can make the promise the reconstruct
|
||
cargo classes make: **bit-identical output on every machine, no tolerance** —
|
||
because its synthesis is pure `i64` arithmetic (integer transposed
|
||
convolutions + arithmetic-shift requantize + integer ReLU), not float ONNX.
|
||
Two sinks honoring the spec produce the same bytes on every architecture.
|
||
|
||
**Provenance.** `gradient`, `bloom` and `facade` are generated here by `wai_int_synth_vectors` from a fixed SplitMix64 sequence. `photo` is not: it codes a third-party picture that was not made for this repository and has no recorded licence for redistribution, and its decoder was trained on images that are not recorded. It is to be replaced by generated content.
|
||
|
||
The float decoders (`bmshj2018`, `mbt2018_mean`, `elic`, `mlicpp`, …) are
|
||
deliberately **not** here: their ONNX synthesis is float, so two conforming
|
||
sinks on different hardware may differ in the low bits — a byte-equality
|
||
corpus would be a false promise for them. The integer decoder is the member
|
||
where `intent = replicate` is sound, so it is the member with a cross-arch
|
||
byte-equality corpus.
|
||
|
||
Because the decoder pulls in no float runtime, it lives at the crate root
|
||
behind the light `neural_int` feature (no `ort`), so this corpus is verified
|
||
in the same no-system-deps CI matrix as the reconstruct classes, which checks
|
||
the integer-exact contract on x86-64 and arm64 (Linux x86_64, Linux arm64 and
|
||
macOS arm64) on every push that touches it.
|
||
|
||
## Layout
|
||
|
||
- `vectors/<name>/int_synth.json` — the integer decoder model (per-layer
|
||
weights/bias as `i32`, stride, requantize shift, integer ReLU flag).
|
||
- `vectors/<name>/latent.bin` — the wire payload (12-byte header +
|
||
channel-major `int8` latents).
|
||
- `sha256.json` — per vector, the SHA-256 of its decoded output as `RGB8`
|
||
(SPEC §7, *Digests*): the conformance digest.
|
||
- `expected_hashes.json` — per vector: `s_log2`, `in_channels`, `layers`,
|
||
`width`, `height`, and `rgb_hash`, a domain-prefixed BLAKE3 of the decoded
|
||
RGB, kept unchanged as a divergence diagnostic.
|
||
|
||
The vectors keep their unpinned `<dir>/model` convention. A sink that pins an
|
||
`int_synth` parameter set pins `int_synth.json`, the one file that holds it
|
||
(SPEC §5).
|
||
|
||
Synthetic vectors (realistic multi-layer decoders, distinct shapes) — model
|
||
weights/latents authored from a fixed integer SplitMix64 sequence, so `gen`
|
||
is reproducible without a float RNG:
|
||
- `gradient` — 8×8 latent, 4 ch → ×2 → ×2 = 38×38 RGB, two layers, one ReLU.
|
||
- `bloom` — 4×4 latent, 8 ch → ×4 (single deep upsample) = 18×18 RGB.
|
||
- `facade` — 16×16 latent, 3 ch → ×2 → ×2 → refine = 72×72 RGB, three
|
||
layers, two ReLUs (a deeper integer stack).
|
||
|
||
And one **real trained model**, on a third-party picture (see *Provenance*, above):
|
||
- `photo` — a 128×128 third-party picture, encoded to a 6,156-byte int8
|
||
latent (96×8×8) + a four-layer integer decoder trained on images the corpus does not record
|
||
(`tools/wai_int_synth_real.py`: float-train + post-training quantization with
|
||
per-layer activation-scale calibration). Reconstruction ≈ 21.6 dB at ~3 bpp.
|
||
This is the honest scope of int_synth: a pure integer transposed-conv decoder
|
||
with no entropy model — its **quality is modest by design**; its **point is
|
||
that the decode is bit-identical on every machine**. At `gen` time the Rust
|
||
decoder is cross-checked against the tool's python integer-forward
|
||
(`ref_rgb.bin`) to max|Δ|=0, and the frozen `rgb_hash` then enforces the same
|
||
bytes on every platform in CI — a real trained model, decoded identically
|
||
everywhere.
|
||
|
||
## Running
|
||
|
||
```
|
||
cd ../wai-rs
|
||
cargo run --features neural_int --bin wai_int_synth_vectors -- verify ../neural-conformance
|
||
cargo run --features neural_int --bin wai_int_synth_vectors -- gen ../neural-conformance # regenerate
|
||
```
|
||
|
||
Pass = decoded-RGB BLAKE3 + dimensions equality, on every machine.
|