wai: neural-conformance's trained vector is trained and coded on generated pictures, and every vector has a licence record #639

Merged
dcharlot merged 3 commits from wai/neural-corpus-licence into main 2026-11-06 14:18:19 -05:00
29 changed files with 884 additions and 138 deletions

View file

@ -168,6 +168,14 @@ jobs:
fi
bash scripts/check-public-names.sh
# ── Every vector of a corpus that keeps licence records has one ──────────
- name: "corpus licences: the check fails closed (self-test)"
if: ${{ !cancelled() }}
run: bash scripts/check-wai-corpus-licences-selftest.sh
- name: "corpus licences: every vector has a record and a licence that permits redistribution"
if: ${{ !cancelled() }}
run: python3 -I scripts/check-wai-corpus-licences.py
# ── No library source file is unreachable from its crate root ───────────
# An orphan `.rs` (one no `mod`, `#[path]` or `include!` names) is never compiled,
# so nothing else in this repository can see it. Both directions, before trusting

View file

@ -114,6 +114,8 @@ on:
- 'wai/wai-web/**'
# The audio provenance check below reads the corpora above with this tool.
- 'wai/tools/wai_audio_provenance_check.py'
# The trained int_synth vector's picture check below runs this tool.
- 'wai/tools/wai_int_synth_real.py'
- '.github/workflows/wai-conformance.yml'
# The steps of the independent verifiers, the corpora of cargo-classes, and the script
# that derives this workflow's lists from them (scripts/wai-conformance-lists.py).
@ -610,6 +612,43 @@ jobs:
else
echo "audio provenance: SKIPPED — no python3 on this host"
fi
# neural-conformance's trained vector codes the picture its tool generates: the tool
# regenerates that picture from the seed in the vector's meta.json, with the Python
# standard library alone, and requires the vector's reference decode, whose SHA-256
# is the corpus's digest for it, to reconstruct the picture at the PSNR meta.json
# records, and the vector's licence record to state the seed and streams meta.json
# records. One leg: the generator is integer arithmetic.
- name: The trained int_synth vector codes the picture its tool generates (macOS)
if: matrix.os == 'macos'
shell: bash
working-directory: wai
run: python3 -I tools/wai_int_synth_real.py --check neural-conformance/vectors/photo
# The same check on a copy of the corpus whose licence record gives the training
# pictures as streams 1 to 64 instead of meta.json's 1 to 512: it must fail.
- name: The trained int_synth vector's check fails on a licence record that misstates its training pictures (macOS)
if: matrix.os == 'macos'
shell: bash
working-directory: wai
run: |
set -uo pipefail
t=$(mktemp -d)
trap 'rm -rf "$t"' EXIT
mkdir -p "$t/vectors"
cp -R neural-conformance/vectors/photo "$t/vectors/"
cp neural-conformance/sha256.json neural-conformance/licences.json "$t/"
python3 -I - "$t/licences.json" <<'P'
import json, sys
d = json.load(open(sys.argv[1]))
tr = d["vectors"]["photo"]["training"]
assert tr["input"].endswith("streams 1 to 512"), tr["input"]
tr["input"] = tr["input"][: -len("512")] + "64"
json.dump(d, open(sys.argv[1], "w"))
P
s=0
python3 -I tools/wai_int_synth_real.py --check "$t/vectors/photo" > "$t/out" 2>&1 || s=$?
cat "$t/out"
test "$s" = 1
grep -qF "streams 1 to 64'; meta.json says" "$t/out"
# OER's second implementation is portable C, not Rust. Rebuild it from source and
# require its output to match the pinned corpus byte for byte: the kernel is
# deterministic, and wai_oer_vectors above has already verified the pinned copy,

View file

@ -174,14 +174,14 @@ jobs:
bounds 0 neural-conformance/vectors/facade/int_synth.json --capability wai.neural.int_synth --profile acc32 \
--check $C/c6a362696d61605931ef4da98ae630362dcd73ea6be7864db6e61d6c5abbcd24.json
bounds 0 neural-conformance/vectors/photo/int_synth.json --capability wai.neural.int_synth --profile acc64 \
--check $C/50868e72f5f1599569c413ed1f67ab8610fa0b3c17919162cbf370f49275125c.json
--check $C/7c15df3437b7f9a47aeec9107c782149e308b72f81c7a3981999769ab9eda1e5.json
# With --witness, a node-local graph for each failing node: photo's node 6, whose
# input attains convT.acc's upper end. With --advise, the clamps that would let
# the overflow set meet acc64.
bounds 1 neural-conformance/vectors/photo/int_synth.json --capability wai.neural.int_synth --profile acc32 \
--witness "$T/witness"
test -s "$T/witness/synth-6.json"
grep -q '"acc":"26172239136"' "$T/witness/synth-6.json"
grep -q '"acc":"3092313901"' "$T/witness/synth-6.json"
bounds 1 ops-conformance/sets/overflow_acc64/int_synth.json --capability wai.neural.int_synth --advise acc64
# A certificate that is not the committed one is exit 3.
bounds 3 neural-conformance/vectors/photo/int_synth.json --capability wai.neural.int_synth \

View file

@ -0,0 +1,123 @@
#!/usr/bin/env bash
#
# check-wai-corpus-licences-selftest.sh — check-wai-corpus-licences.py can fail.
#
# A check that cannot fail is decoration. This builds a scratch tree shaped like the
# repository (wai/neural-conformance with vector directories, its two hash manifests and
# licences.json, and a generator below wai/) whose every vector has a licence record, and
# asserts that the check passes it, and fails each way a record can be missing, name a
# licence that does not permit redistribution, or leave out what its content needs.
# Nothing here reads the real tree, and nothing needs a network.
set -uo pipefail
here=$(cd "$(dirname "$0")" && pwd)
check="$here/check-wai-corpus-licences.py"
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
fails=0
# mk <root>: a corpus that holds. `seeded` and `trained` are generated, `third` is a
# third-party picture under CC-BY-4.0; `trained` is marked trained in expected_hashes.json.
mk() {
local c="$1/wai/neural-conformance"
mkdir -p "$c/vectors/seeded" "$c/vectors/trained" "$c/vectors/third" "$1/wai/tools"
printf '# a generator\n' > "$1/wai/tools/gen.py"
cat > "$c/expected_hashes.json" <<'J'
{"seeded": {"rgb_hash": "00"}, "trained": {"trained": true, "rgb_hash": "01"}, "third": {"rgb_hash": "02"}}
J
cat > "$c/sha256.json" <<'J'
{"seeded": {"sha256": "00"}, "trained": {"sha256": "01"}, "third": {"sha256": "02"}}
J
cat > "$c/licences.json" <<'J'
{"format": "wai-corpus-licences/1", "vectors": {
"seeded": {"licence": "Apache-2.0", "content": "generated", "generator": "tools/gen.py", "input": "seed 1"},
"trained": {"licence": "Apache-2.0", "content": "generated", "generator": "tools/gen.py", "input": "seed 2, stream 0",
"training": {"content": "generated", "generator": "tools/gen.py", "input": "seed 2, streams 1 to 8"}},
"third": {"licence": "CC-BY-4.0", "content": "third-party", "source": "a picture and its licence page", "generator": "tools/gen.py"}
}}
J
}
# edit <root> <python statement on `d`, the parsed licences.json>
edit() {
python3 -I - "$1/wai/neural-conformance/licences.json" "$2" <<'P'
import json, sys
p, stmt = sys.argv[1], sys.argv[2]
d = json.load(open(p))
v = d["vectors"]
exec(stmt)
json.dump(d, open(p, "w"))
P
}
want() { # want <expect-status> <root> <label> [<text the output must contain>]
python3 -I "$check" --root "$2" > "$tmp/out" 2>&1
got=$?
if [ "$got" != "$1" ]; then
echo "FAIL: $3 — expected exit $1, got $got"
sed 's/^/ /' "$tmp/out"
fails=$((fails + 1))
elif [ -n "${4:-}" ] && ! grep -qF -- "$4" "$tmp/out"; then
echo "FAIL: $3 — the output does not say: $4"
sed 's/^/ /' "$tmp/out"
fails=$((fails + 1))
else
echo "ok: $3"
fi
}
case_() { # case_ <label> <python statement> <text the finding must contain>
local r="$tmp/$((n += 1))"
mk "$r"
edit "$r" "$2"
want 1 "$r" "$1" "$3"
}
n=0
r="$tmp/ok"; mk "$r"
want 0 "$r" "passes when every vector has a record" "wai/neural-conformance: CC-BY-4.0: third"
r="$tmp/nofile"; mk "$r"; rm "$r/wai/neural-conformance/licences.json"
want 1 "$r" "fails when the corpus has no licences.json" "no licences.json"
r="$tmp/badjson"; mk "$r"; printf '{"format": ' > "$r/wai/neural-conformance/licences.json"
want 1 "$r" "fails when licences.json does not parse" "does not parse"
r="$tmp/newdir"; mk "$r"; mkdir -p "$r/wai/neural-conformance/vectors/added"
want 1 "$r" "fails on a vector directory with no record" "vector added has no licence record"
r="$tmp/newhash"; mk "$r"
python3 -I -c 'import json,sys; p=sys.argv[1]; d=json.load(open(p)); d["listed"]={"rgb_hash":"03"}; json.dump(d,open(p,"w"))' \
"$r/wai/neural-conformance/expected_hashes.json"
want 1 "$r" "fails on a manifest entry with no record" "vector listed has no licence record"
case_ "fails on the wrong format" 'd["format"] = "wai-corpus-licences/0"' "not a wai-corpus-licences/1 document"
case_ "fails on a record with no vector" 'v["gone"] = dict(v["seeded"])' "gone: no directory vectors/gone"
case_ "fails on a record with no licence" 'del v["seeded"]["licence"]' "seeded: licence None is not one of"
case_ "fails on a licence that grants nothing" 'v["seeded"]["licence"] = "NOASSERTION"' "licence 'NOASSERTION' is not one of"
case_ "fails on a licence that is not SPDX's spelling" 'v["seeded"]["licence"] = "apache 2.0"' "licence 'apache 2.0' is not one of"
case_ "fails on unknown content" 'v["seeded"]["content"] = "synthetic"' "content 'synthetic' is neither"
case_ "fails on generated content with no generator" 'del v["seeded"]["generator"]' "generated content names no generator"
case_ "fails on a generator that does not exist" 'v["seeded"]["generator"] = "tools/missing.py"' "generator 'tools/missing.py' is not a file below wai/"
case_ "fails on generated content with no input" 'v["seeded"]["input"] = " "' "generated content names no input"
case_ "fails on generated content with a source" 'v["seeded"]["source"] = "a photograph"' "generated content has a source"
case_ "fails on third-party content with no source" 'del v["third"]["source"]' "third-party content names no source"
case_ "fails on a trained model with no training record" 'del v["trained"]["training"]' "a trained model with no training record"
case_ "fails on a training record on a model not marked trained" 'v["seeded"]["training"] = dict(v["trained"]["training"])' "a training record on a vector whose model is not marked trained"
case_ "fails on training data with no generator" 'del v["trained"]["training"]["generator"]' "trained training: generated content names no generator"
case_ "fails on third-party training data with no licence" \
'v["trained"]["training"] = {"content": "third-party", "source": "photographs"}' "trained training: licence None is not one of"
case_ "fails on third-party training data under a licence that grants nothing" \
'v["trained"]["training"] = {"content": "third-party", "source": "photographs", "licence": "unknown"}' "licence 'unknown' is not one of"
case_ "fails on an unknown member" 'v["seeded"]["licnece"] = "Apache-2.0"' "unknown member 'licnece'"
case_ "fails on an unknown member of a training record" 'v["trained"]["training"]["seed"] = 2' "trained training: unknown member 'seed'"
r="$tmp/tp-train"; mk "$r"
edit "$r" 'v["trained"]["training"] = {"content": "third-party", "source": "photographs and their licence pages", "licence": "CC0-1.0"}'
want 0 "$r" "passes third-party training data under a licence that permits redistribution"
if [ "$fails" != 0 ]; then
echo "check-wai-corpus-licences-selftest: $fails case(s) failed."
exit 1
fi
echo "check-wai-corpus-licences-selftest: all cases pass."

View file

@ -0,0 +1,171 @@
#!/usr/bin/env python3
"""check-wai-corpus-licences.py: every vector of a WAI corpus that keeps licence records
has one, and every record states a licence that permits redistribution.
A corpus listed in CORPORA keeps `licences.json` beside its README:
{"format": "wai-corpus-licences/1",
"vectors": {"<name>": {"licence": "<SPDX id>", "content": "generated" | "third-party",
"generator": "<path below wai/>", "input": "<what it reads>",
"source": "<where a third-party input comes from>",
"training": {"content": ..., "generator": ..., "input": ...,
"source": ..., "licence": ...}}}}
What each corpus is held to:
1. `licences.json` exists, parses, and has this format.
2. Every vector has a record: each directory under `vectors/`, and each name in the
corpus's `expected_hashes.json` and `sha256.json`, is a key of `vectors`. Every key
names a directory under `vectors/`.
3. `licence` is one of LICENCES, the licences that permit redistribution.
4. `content` "generated" names a `generator` that exists below `wai/` and a non-empty
`input`, and no `source`. `content` "third-party" names a non-empty `source`; its
`licence` is the source's, and any `generator` it names exists.
5. A vector whose `expected_hashes.json` entry has `"trained": true` has a `training`
record, held to rule 4, whose `licence`, required when its content is third-party,
is one of LICENCES; a vector not marked trained has none.
6. A record has no member other than these.
The check holds what the records state, not their truth: that is a reviewer's, and for
`neural-conformance`'s trained vector also `tools/wai_int_synth_real.py --check`'s.
check-wai-corpus-licences.py [--root DIR]
Exit 0 when every corpus holds, printing each corpus's vectors by licence; 1 with one
line a finding otherwise. Standard library only.
"""
import json
import os
import sys
# The corpora that keep a licence record for each vector, relative to the root.
CORPORA = ["wai/neural-conformance"]
FORMAT = "wai-corpus-licences/1"
# SPDX identifiers of the licences a committed vector, or the data a committed model was
# trained on, may carry: each permits redistribution.
LICENCES = {"Apache-2.0", "BSD-2-Clause", "BSD-3-Clause", "CC-BY-4.0", "CC0-1.0", "MIT"}
VECTOR_KEYS = {"licence", "content", "generator", "input", "source", "training"}
TRAINING_KEYS = {"content", "generator", "input", "source", "licence"}
def names_in(path):
"""The top-level keys of a JSON object file, or None when the file does not exist."""
if not os.path.exists(path):
return None
with open(path, encoding="utf-8") as fh:
doc = json.load(fh)
return doc if isinstance(doc, dict) else {}
def nonempty(v):
return isinstance(v, str) and v.strip() != ""
def check_content(root, where, rec, findings):
"""Rule 4, for a vector record or a training record."""
content = rec.get("content")
gen = rec.get("generator")
if gen is not None and not (nonempty(gen) and os.path.isfile(os.path.join(root, "wai", gen))):
findings.append(f"{where}: generator {gen!r} is not a file below wai/")
if content == "generated":
if gen is None:
findings.append(f"{where}: generated content names no generator")
if not nonempty(rec.get("input")):
findings.append(f"{where}: generated content names no input")
if "source" in rec:
findings.append(f"{where}: generated content has a source")
elif content == "third-party":
if not nonempty(rec.get("source")):
findings.append(f"{where}: third-party content names no source")
else:
findings.append(f"{where}: content {content!r} is neither \"generated\" nor \"third-party\"")
def check_corpus(root, corpus, findings, split):
base = os.path.join(root, corpus)
path = os.path.join(base, "licences.json")
if not os.path.isfile(path):
findings.append(f"{corpus}: no licences.json")
return
try:
with open(path, encoding="utf-8") as fh:
doc = json.load(fh)
except (OSError, ValueError) as e:
findings.append(f"{corpus}/licences.json: does not parse: {e}")
return
if not isinstance(doc, dict) or doc.get("format") != FORMAT or not isinstance(doc.get("vectors"), dict):
findings.append(f"{corpus}/licences.json: not a {FORMAT} document with a `vectors` object")
return
records = doc["vectors"]
vdir = os.path.join(base, "vectors")
dirs = sorted(d for d in os.listdir(vdir) if os.path.isdir(os.path.join(vdir, d))) if os.path.isdir(vdir) else []
try:
hashes = names_in(os.path.join(base, "expected_hashes.json")) or {}
sha = names_in(os.path.join(base, "sha256.json")) or {}
except ValueError as e:
findings.append(f"{corpus}: a hash manifest does not parse: {e}")
return
for name in sorted(set(dirs) | set(hashes) | set(sha)):
if name not in records:
findings.append(f"{corpus}: vector {name} has no licence record")
for name in sorted(records):
where = f"{corpus}/licences.json: {name}"
rec = records[name]
if name not in dirs:
findings.append(f"{where}: no directory vectors/{name}")
if not isinstance(rec, dict):
findings.append(f"{where}: not an object")
continue
for k in sorted(set(rec) - VECTOR_KEYS):
findings.append(f"{where}: unknown member {k!r}")
lic = rec.get("licence")
if lic not in LICENCES:
findings.append(f"{where}: licence {lic!r} is not one of {', '.join(sorted(LICENCES))}")
check_content(root, where, rec, findings)
trained = isinstance(hashes.get(name), dict) and hashes[name].get("trained") is True
tr = rec.get("training")
if trained and tr is None:
findings.append(f"{where}: a trained model with no training record")
elif not trained and tr is not None:
findings.append(f"{where}: a training record on a vector whose model is not marked trained")
if isinstance(tr, dict):
tw = where + " training"
for k in sorted(set(tr) - TRAINING_KEYS):
findings.append(f"{tw}: unknown member {k!r}")
check_content(root, tw, tr, findings)
if (tr.get("content") == "third-party" or "licence" in tr) and tr.get("licence") not in LICENCES:
findings.append(f"{tw}: licence {tr.get('licence')!r} is not one of {', '.join(sorted(LICENCES))}")
elif tr is not None:
findings.append(f"{where}: training is not an object")
if lic in LICENCES:
split.setdefault(corpus, {}).setdefault(lic, []).append(name)
def main(argv):
root = "."
args = argv[1:]
if args[:1] == ["--root"] and len(args) == 2:
root = args[1]
elif args:
print("usage: check-wai-corpus-licences.py [--root DIR]", file=sys.stderr)
return 2
findings, split = [], {}
for corpus in CORPORA:
check_corpus(root, corpus, findings, split)
for f in findings:
print(f)
if findings:
print(f"check-wai-corpus-licences: {len(findings)} finding(s)")
return 1
for corpus in CORPORA:
for lic, names in sorted(split.get(corpus, {}).items()):
print(f"{corpus}: {lic}: {', '.join(names)}")
print("check-wai-corpus-licences: every vector has a licence record")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))

View file

@ -86,6 +86,15 @@ provenance line. `wai-rs/tests/corpus_provenance.rs` requires the line in every
corpus, and `tools/wai_audio_provenance_check.py`, which CI runs, fails when a
committed audio vector correlates with a reference recording the host has.
A corpus can also keep a licence record for each vector, in a `licences.json` beside
its README: the vector's licence, as an SPDX identifier from a list of licences that
permit redistribution; whether its content is generated here or a third-party input;
the generator and its input, or the source; and, for a vector whose model was trained,
the same of its training data. `neural-conformance` keeps one.
`scripts/check-wai-corpus-licences.py`, run on every change, fails on a vector of such a
corpus that has no record, and on a record whose licence is outside that list or that
leaves out what its content needs.
### Independent implementations
Conformance is defined by the committed corpora (`*-conformance/`) and the

View file

@ -0,0 +1,59 @@
### neural-conformance: the trained vector is trained and coded on generated pictures, and every vector has a licence record
- **`photo` is generated content.** It coded a third-party picture with no recorded licence for
redistribution, and its decoder was trained on pictures the corpus did not record. Now its decoder is
trained on, and its latent codes, dead-leaves pictures that `tools/wai_int_synth_real.py` generates
from a seed. The vector keeps its name, its shape (a 96 × 8 × 8 int8 latent, a four-layer decoder,
128 × 128 RGB), its payload format and its capability; its weights, latent and digests are new.
- **`tools/wai_int_synth_real.py` reads no image file.** `picture()` writes each picture from a seed in
integer arithmetic only, a SplitMix64 stream per picture, so a seed gives the same pictures on every
machine and every Python: a vertical gradient between two colours, 400 discs painted back to front with
power-law radii, colours and a linear shading, then noise of ±2. The coded picture is stream 0 of seed
20261104 and the training pictures are streams 1 to 512, so the decoder never sees the picture it
codes. Training runs on the CPU, seeded from the same seed, with deterministic algorithms, four threads
and seeded batches; two runs on one host gave byte-identical files. Its loss is taken before the
output's clamp: clamped, a decoder whose first outputs fall below 0 gets no gradient and stays black,
which it did on these pictures. Each layer's weight scale is now chosen so that its shift carries its
calibrated activation scale exactly; rounding the shift alone left each layer's gain off by up to √2.
Two runs with the same settings, whose float decoders both reconstruct the coded picture at 18.46 dB,
lost 3.73 dB in the integer decode the earlier way and 0.44 dB this way.
`--picture-only` writes one picture, and `--check` regenerates the picture a vector codes from the seed
in its `meta.json` and requires its reference decode, whose SHA-256 must be the corpus's digest for the
vector, to reconstruct it at the PSNR `meta.json` records: 18.02 dB, where the same decode scores 8.5
to 9.5 dB against the training pictures of streams 1 to 5. `--check` also requires the vector's record
in `licences.json` to state the seed, the coded picture's stream and the training streams `meta.json`
records, and those training streams to leave out the coded picture's. `--picture-only` and `--check`
need only the Python standard library. CI runs `--check` on the macOS leg of the conformance
workflow, and the step after it runs it on a copy whose record gives the training pictures as streams
1 to 64 and requires it to fail.
- **Regenerated through their generators:** `vectors/photo/` (`int_synth.json`, `latent.bin`,
`ref_rgb.bin`, `meta.json`) by the tool; `expected_hashes.json` and `sha256.json` by
`wai_int_synth_vectors gen`, which checked the Rust decode against the tool's integer forward to
max|Δ| = 0 and left `gradient`, `bloom` and `facade` unchanged; the photo set's certificate and lowered
graph in `ops-conformance/certificates/` and `index.json`, whose note for the set now says how it was
made, by `wai-verify vectors gen`; and the demo's byte copies in `wai-web/demo/int_synth/`, whose
page's menu and README now give the vector's PSNR and say its picture is generated. The set
still meets `acc64` and fails `acc32` at node 6, `convT.acc`; the wai-ops workflow names its new
certificate and the new accumulator of the node's witness.
- **Licence records.** `neural-conformance/licences.json` (`wai-corpus-licences/1`) records each
vector's licence as an SPDX identifier, whether its content is generated here or a third-party input,
the generator and its input or the source, and, for a trained vector, the same of its training data.
Every vector is Apache-2.0. `scripts/check-wai-corpus-licences.py`, a step of the repository checks on
every change, fails on a vector without a record (a directory under `vectors/` or a name in either hash
manifest), a record for no vector, a licence outside its list of licences that permit redistribution,
a record that leaves out what its content needs, a trained vector without a training record, and an
unknown member. `scripts/check-wai-corpus-licences-selftest.sh`, the step before it, builds a scratch
corpus and shows the check fails each of those ways. CONTRIBUTING (*Corpus content*) states the record.
- **Records kept.** The sink-energy tables measured on the earlier `photo` name its latent by its SHA-256
in their `procedure.txt`; they are records of those runs and are unchanged. `int_video`'s `video_real`
is in another corpus and is unchanged; `wai-rs/tests/vectors/README.md` still states that it codes a
third-party picture.
- **Breaking:** `wire:corpus:wai/neural-conformance/expected_hashes.json`: the `photo` vector is retrained and recoded on generated pictures, so a reader built against the old vector's bytes, weights, digests or PSNR disagrees with the new one.
- **Breaking:** `wire:corpus:wai/neural-conformance/sha256.json`: the `photo` vector is retrained and recoded on generated pictures, so a reader built against the old vector's bytes, weights, digests or PSNR disagrees with the new one.
- **Breaking:** `wire:corpus:wai/neural-conformance/vectors/photo/int_synth.json`: the `photo` vector is retrained and recoded on generated pictures, so a reader built against the old vector's bytes, weights, digests or PSNR disagrees with the new one.
- **Breaking:** `wire:corpus:wai/neural-conformance/vectors/photo/latent.bin`: the `photo` vector is retrained and recoded on generated pictures, so a reader built against the old vector's bytes, weights, digests or PSNR disagrees with the new one.
- **Breaking:** `wire:corpus:wai/neural-conformance/vectors/photo/meta.json`: the `photo` vector is retrained and recoded on generated pictures, so a reader built against the old vector's bytes, weights, digests or PSNR disagrees with the new one.
- **Breaking:** `wire:corpus:wai/neural-conformance/vectors/photo/ref_rgb.bin`: the `photo` vector is retrained and recoded on generated pictures, so a reader built against the old vector's bytes, weights, digests or PSNR disagrees with the new one.
- **Breaking:** `wire:corpus:wai/ops-conformance/certificates/index.json`: the operator certificate for `wai.neural.int_synth` certifies the retrained decoder, so its digest, its record and its packed weights are new.
- **Breaking:** `wire:corpus:wai/ops-conformance/certificates/wai.neural.int_synth/50868e72f5f1599569c413ed1f67ab8610fa0b3c17919162cbf370f49275125c.json`: the operator certificate for `wai.neural.int_synth` certifies the retrained decoder, so its digest, its record and its packed weights are new.
- **Breaking:** `wire:corpus:wai/ops-conformance/certificates/wai.neural.int_synth/50868e72f5f1599569c413ed1f67ab8610fa0b3c17919162cbf370f49275125c/synth.whg1`: the operator certificate for `wai.neural.int_synth` certifies the retrained decoder, so its digest, its record and its packed weights are new.

View file

@ -7,7 +7,7 @@ because its synthesis is pure `i64` arithmetic (integer transposed
convolutions + arithmetic-shift requantize + integer ReLU), not float ONNX.
Two sinks honoring the spec produce the same bytes on every architecture.
**Provenance.** `gradient`, `bloom` and `facade` are generated here by `wai_int_synth_vectors` from a fixed SplitMix64 sequence. `photo` is not: it codes a third-party picture that was not made for this repository and has no recorded licence for redistribution, and its decoder was trained on images that are not recorded. It is to be replaced by generated content.
**Provenance.** Generated here. `gradient`, `bloom` and `facade` are written by `wai_int_synth_vectors` from a fixed SplitMix64 sequence. `photo`'s decoder is trained on dead-leaves pictures that `tools/wai_int_synth_real.py` generates from a seed, and its latent codes another picture of the same seed; the tool reads no image file. Every vector is Apache-2.0, and `licences.json` records each one's licence (*Licences*, below).
The float decoders (`bmshj2018`, `mbt2018_mean`, `elic`, `mlicpp`, …) are
deliberately **not** here: their ONNX synthesis is float, so two conforming
@ -33,6 +33,12 @@ macOS arm64) on every push that touches it.
- `expected_hashes.json` — per vector: `s_log2`, `in_channels`, `layers`,
`width`, `height`, and `rgb_hash`, a domain-prefixed BLAKE3 of the decoded
RGB, kept unchanged as a divergence diagnostic.
- `vectors/photo/ref_rgb.bin` and `vectors/photo/meta.json` — the trained
vector's reference decode, written by the tool's integer forward, and how it
was made: its shape, payload size, PSNR and bits per pixel, the seed and
streams of its pictures, and the training run's settings.
- `licences.json` — per vector, its licence and where its content comes from
(*Licences*, below).
The vectors keep their unpinned `<dir>/model` convention. A sink that pins an
`int_synth` parameter set pins `int_synth.json`, the one file that holds it
@ -46,19 +52,81 @@ is reproducible without a float RNG:
- `facade` — 16×16 latent, 3 ch → ×2 → ×2 → refine = 72×72 RGB, three
layers, two ReLUs (a deeper integer stack).
And one **real trained model**, on a third-party picture (see *Provenance*, above):
- `photo` — a 128×128 third-party picture, encoded to a 6,156-byte int8
latent (96×8×8) + a four-layer integer decoder trained on images the corpus does not record
(`tools/wai_int_synth_real.py`: float-train + post-training quantization with
per-layer activation-scale calibration). Reconstruction ≈ 21.6 dB at ~3 bpp.
And one **trained model**, on pictures generated here (see *Provenance*, above):
- `photo` — a 128×128 dead-leaves picture, encoded to a 6,156-byte int8
latent (96×8×8) + a four-layer integer decoder (`tools/wai_int_synth_real.py`:
float training, then post-training quantization with per-layer
activation-scale calibration, each layer's weight scale chosen so that its
shift carries its activation scale exactly). The decoder is trained on 512 other
pictures of the same generator and never sees the one it codes.
Reconstruction 18.02 dB at 3 bpp (the float decoder's is 18.46 dB).
This is the honest scope of int_synth: a pure integer transposed-conv decoder
with no entropy model — its **quality is modest by design**; its **point is
that the decode is bit-identical on every machine**. At `gen` time the Rust
decoder is cross-checked against the tool's python integer-forward
(`ref_rgb.bin`) to max|Δ|=0, and the frozen `rgb_hash` then enforces the same
bytes on every platform in CI — a real trained model, decoded identically
bytes on every platform in CI — a trained model, decoded identically
everywhere.
### How `photo` was made
The pictures are dead leaves, written by the tool's `picture()` from a seed in
integer arithmetic only (a SplitMix64 stream per picture), so a seed gives the
same pictures on every machine and every Python: a vertical gradient between
two colours, then 400 discs painted back to front, each with a uniform centre,
a radius ⌊32·16 / (16 + u)⌋ (at least 2) for u uniform in 0..239, a colour
uniform in 16..239 per channel and a linear shading of slope up to 3/8 a pixel,
then noise uniform in −2..2 on every sample. The seed is **20261104**: stream 0
is the picture the vector codes, streams 1 to 512 the training pictures.
From `wai/`, with torch 2.14.1 (CPU build) and numpy 2.5.4 on an x86_64 Linux host:
python tools/wai_int_synth_real.py --out <dir>
then `int_synth.json`, `latent.bin`, `ref_rgb.bin` and `meta.json` copied from
`<dir>` into `vectors/photo/`, and `wai_int_synth_vectors gen` run (*Running*,
below). The command's defaults are the committed vector's: seed 20261104,
512 training pictures, 3,000 Adam steps on batches of 32 drawn by a generator
seeded from the seed, a learning rate of 2·10⁻⁴ that falls to a tenth for the
last quarter of the steps, four threads on the CPU, deterministic algorithms.
Two runs on that host gave byte-identical files. A different host or torch
build may train to different weights; the committed files are what
conformance checks, and their decode is integer and the same everywhere.
The pictures alone need only the Python standard library:
python3 tools/wai_int_synth_real.py --picture-only photo.ppm --stream 0
and so does the check that the vector codes the picture its seed generates:
it regenerates that picture, requires `ref_rgb.bin`'s SHA-256 to be the
vector's digest in `sha256.json`, and requires `ref_rgb.bin` to reconstruct the
picture at the PSNR `meta.json` records (the same decode scores 8.5 to 9.5 dB
against the pictures of streams 1 to 5). It also requires `photo`'s record in
`licences.json` to state the seed, the coded picture's stream and the training
streams `meta.json` records, and the training streams to leave out the coded
picture's. CI runs it on the macOS leg of the conformance workflow:
python3 tools/wai_int_synth_real.py --check neural-conformance/vectors/photo
## Licences
`licences.json` (`wai-corpus-licences/1`) holds a record for each vector: its
licence, as an SPDX identifier; `content`, `generated` (with the `generator`
below `wai/` and its `input`) or `third-party` (with its `source`); and, for a
vector whose model was trained, a `training` record of the same kind for its
training data. By licence:
- **Apache-2.0** — `bloom`, `facade`, `gradient` and `photo`, all generated
here; `photo`'s training data is generated here too.
`scripts/check-wai-corpus-licences.py`, run on every change by the repository
checks, fails on a vector of this corpus without a record (a directory under
`vectors/` or a name in either hash manifest), on a record for no vector, on a
licence outside its list of licences that permit redistribution, on a record
that leaves out what its content needs, and on a trained vector without a
training record. `scripts/check-wai-corpus-licences-selftest.sh` shows it fails
each of those ways.
## Running
```
@ -68,3 +136,9 @@ cargo run --features neural_int --bin wai_int_synth_vectors -- gen ../neural-
```
Pass = decoded-RGB BLAKE3 + dimensions equality, on every machine.
From the repository root, the licence records:
```
python3 scripts/check-wai-corpus-licences.py
```

View file

@ -28,8 +28,8 @@
"width": 128,
"height": 128,
"layers": 4,
"rgb_hash": "1677d7112db6cadd70b0eeba8df263f9605cf3875809ce4b26e9c914f62fb582",
"psnr_db": 21.6,
"rgb_hash": "093d6c2d2670ecd23b2d338fcec2f49361098692a671429435f11ff50908c79d",
"psnr_db": 18.02,
"bpp": 3.006,
"in_channels": 96,
"s_log2": 3

View file

@ -0,0 +1,34 @@
{
"format": "wai-corpus-licences/1",
"vectors": {
"bloom": {
"licence": "Apache-2.0",
"content": "generated",
"generator": "wai-rs/src/bin/wai_int_synth_vectors.rs",
"input": "a SplitMix64 sequence from seed 0xC0FFEE42: the decoder's integer weights, then the latent"
},
"facade": {
"licence": "Apache-2.0",
"content": "generated",
"generator": "wai-rs/src/bin/wai_int_synth_vectors.rs",
"input": "a SplitMix64 sequence from seed 0x5EED1010: the decoder's integer weights, then the latent"
},
"gradient": {
"licence": "Apache-2.0",
"content": "generated",
"generator": "wai-rs/src/bin/wai_int_synth_vectors.rs",
"input": "a SplitMix64 sequence from seed 0x12345678: the decoder's integer weights, then the latent"
},
"photo": {
"licence": "Apache-2.0",
"content": "generated",
"generator": "tools/wai_int_synth_real.py",
"input": "the dead-leaves picture the tool generates from seed 20261104, stream 0",
"training": {
"content": "generated",
"generator": "tools/wai_int_synth_real.py",
"input": "the dead-leaves pictures the tool generates from seed 20261104, streams 1 to 512"
}
}
}
}

View file

@ -13,6 +13,6 @@
},
"photo": {
"buffer": "RGB8: rows top to bottom, pixels left to right, R, G, B bytes",
"sha256": "2a3db89614cd632829d873cf59d6300b0e49c5647a9ede1115719a6604d42115"
"sha256": "989168de720dfa6170409721619b451871e17deeae4b2ecde8c96cb499695a0f"
}
}

File diff suppressed because one or more lines are too long

View file

@ -4,6 +4,18 @@
"in_channels": 96,
"s_log2": 3,
"payload_bytes": 6156,
"psnr_db": 21.6,
"bpp": 3.006
}
"psnr_db": 18.02,
"bpp": 3.006,
"seed": 20261104,
"stream": 0,
"training_streams": [
1,
512
],
"steps": 3000,
"batch": 32,
"lr": 0.0002,
"threads": 4,
"torch": "2.14.1+cpu",
"numpy": "2.5.4"
}

File diff suppressed because one or more lines are too long

View file

@ -1 +0,0 @@
{"capability":"wai.neural.int_synth","certificate":"wai-ops/1 bounds","graphs":[{"graph_sha256":"0424c3e9fc68733885cd528542317494c1debba38ce507b31d23590fcc70b102","inputs":[{"hi":"127","kind":"data","lo":"-128","max_h":"0","max_w":"0","policy":"check","tensor":"0"}],"name":"synth","nodes":[{"acc32_params":[],"fits":["acc32","acc64"],"index":"0","intermediates":{"convT.acc":{"bits":"22","hi":"1486634","lo":"-1485967"},"convT.term":{"bits":"15","hi":"16256","lo":"-16129"},"rq.p":{"bits":"22","hi":"1486634","lo":"-1485967"},"rq.q":{"bits":"13","hi":"2903","lo":"-2903"},"rq.t":{"bits":"22","hi":"1486634","lo":"-1485967"},"rq.y":{"bits":"13","hi":"2903","lo":"-2903"}},"k_c":"132104","mac":{"a_bits":["8"],"w_bits":"8"},"npu_flags":[],"op":"conv_transpose","out":[{"bits":"13","hi":"2903","lo":"-2903","policy":"derived","result":{"bits":"13","hi":"2903","lo":"-2903"},"tensor":"1"}],"saturated":false},{"acc32_params":[],"fits":["acc32","acc64"],"index":"1","intermediates":{},"npu_flags":[],"op":"relu","out":[{"bits":"13","hi":"2903","lo":"0","policy":"derived","result":{"bits":"13","hi":"2903","lo":"0"},"tensor":"2"}],"saturated":false},{"acc32_params":[],"fits":["acc32","acc64"],"index":"2","intermediates":{"convT.acc":{"bits":"25","hi":"6369182","lo":"-16550003"},"convT.term":{"bits":"20","hi":"223531","lo":"-368681"},"rq.p":{"bits":"25","hi":"6369182","lo":"-16550003"},"rq.q":{"bits":"20","hi":"199036","lo":"-517188"},"rq.t":{"bits":"25","hi":"6369182","lo":"-16550003"},"rq.y":{"bits":"20","hi":"199036","lo":"-517188"}},"k_c":"5824","mac":{"a_bits":["13"],"w_bits":"8"},"npu_flags":[],"op":"conv_transpose","out":[{"bits":"20","hi":"199036","lo":"-517188","policy":"derived","result":{"bits":"20","hi":"199036","lo":"-517188"},"tensor":"3"}],"saturated":false},{"acc32_params":[],"fits":["acc32","acc64"],"index":"3","intermediates":{},"npu_flags":[],"op":"relu","out":[{"bits":"19","hi":"199036","lo":"0","policy":"derived","result":{"bits":"19","hi":"199036","lo":"0"},"tensor":"4"}],"saturated":false},{"acc32_params":[],"fits":["acc32","acc64"],"index":"4","intermediates":{"convT.acc":{"bits":"30","hi":"189482281","lo":"-488832473"},"convT.term":{"bits":"26","hi":"6369152","lo":"-25277572"},"rq.p":{"bits":"30","hi":"189482281","lo":"-488832473"},"rq.q":{"bits":"26","hi":"11842642","lo":"-30552030"},"rq.t":{"bits":"30","hi":"189482281","lo":"-488832473"},"rq.y":{"bits":"26","hi":"11842642","lo":"-30552030"}},"k_c":"84","mac":{"a_bits":["19"],"w_bits":"8"},"npu_flags":[],"op":"conv_transpose","out":[{"bits":"26","hi":"11842642","lo":"-30552030","policy":"derived","result":{"bits":"26","hi":"11842642","lo":"-30552030"},"tensor":"5"}],"saturated":false},{"acc32_params":[],"fits":["acc32","acc64"],"index":"5","intermediates":{},"npu_flags":[],"op":"relu","out":[{"bits":"25","hi":"11842642","lo":"0","policy":"derived","result":{"bits":"25","hi":"11842642","lo":"0"},"tensor":"6"}],"saturated":false},{"acc32_params":[],"fits":["acc64"],"index":"6","intermediates":{"convT.acc":{"bits":"36","hi":"26172239136","lo":"-25165614250"},"convT.term":{"bits":"32","hi":"1373746472","lo":"-1504015534"},"rq.p":{"bits":"36","hi":"26172239136","lo":"-25165614250"},"rq.q":{"bits":"32","hi":"1635764946","lo":"-1572850891"},"rq.t":{"bits":"36","hi":"26172239136","lo":"-25165614250"},"rq.y":{"bits":"32","hi":"1635764946","lo":"-1572850891"}},"k_c":"1","mac":{"a_bits":["25"],"w_bits":"8"},"npu_flags":[],"op":"conv_transpose","out":[{"bits":"32","hi":"1635764946","lo":"-1572850891","policy":"derived","result":{"bits":"32","hi":"1635764946","lo":"-1572850891"},"tensor":"7"}],"saturated":false},{"acc32_params":[],"fits":["acc32","acc64"],"index":"7","intermediates":{},"npu_flags":[],"op":"clamp","out":[{"bits":"9","hi":"255","lo":"0","policy":"derived","result":{"bits":"9","hi":"255","lo":"0"},"tensor":"8"}],"saturated":false}],"scalars":[]}],"lowering":"int_synth/1","set_sha256":"50868e72f5f1599569c413ed1f67ab8610fa0b3c17919162cbf370f49275125c","verdict":{"acc32":false,"acc32_first_failure":{"bits":"36","graph":"synth","index":"6","key":"convT.acc","where":"node"},"acc64":true,"npu":false,"npu_flags":[],"npu_reasons":["acc32"]}}

View file

@ -0,0 +1 @@
{"capability":"wai.neural.int_synth","certificate":"wai-ops/1 bounds","graphs":[{"graph_sha256":"868097d9a15231e19d2b18183a9873d51ee3a9583e5156ea3e48ce29fb49cd46","inputs":[{"hi":"127","kind":"data","lo":"-128","max_h":"0","max_w":"0","policy":"check","tensor":"0"}],"name":"synth","nodes":[{"acc32_params":[],"fits":["acc32","acc64"],"index":"0","intermediates":{"convT.acc":{"bits":"22","hi":"1131943","lo":"-1131995"},"convT.term":{"bits":"15","hi":"12827","lo":"-12928"},"rq.p":{"bits":"22","hi":"1131943","lo":"-1131995"},"rq.q":{"bits":"14","hi":"4421","lo":"-4422"},"rq.t":{"bits":"22","hi":"1131943","lo":"-1131995"},"rq.y":{"bits":"14","hi":"4421","lo":"-4422"}},"k_c":"166111","mac":{"a_bits":["8"],"w_bits":"8"},"npu_flags":[],"op":"conv_transpose","out":[{"bits":"14","hi":"4421","lo":"-4422","policy":"derived","result":{"bits":"14","hi":"4421","lo":"-4422"},"tensor":"1"}],"saturated":false},{"acc32_params":[],"fits":["acc32","acc64"],"index":"1","intermediates":{},"npu_flags":[],"op":"relu","out":[{"bits":"14","hi":"4421","lo":"0","policy":"derived","result":{"bits":"14","hi":"4421","lo":"0"},"tensor":"2"}],"saturated":false},{"acc32_params":[],"fits":["acc32","acc64"],"index":"2","intermediates":{"convT.acc":{"bits":"25","hi":"10407093","lo":"-12533535"},"convT.term":{"bits":"20","hi":"322733","lo":"-225471"},"rq.p":{"bits":"25","hi":"10407093","lo":"-12533535"},"rq.q":{"bits":"17","hi":"40652","lo":"-48960"},"rq.t":{"bits":"25","hi":"10407093","lo":"-12533535"},"rq.y":{"bits":"17","hi":"40652","lo":"-48960"}},"k_c":"6654","mac":{"a_bits":["14"],"w_bits":"8"},"npu_flags":[],"op":"conv_transpose","out":[{"bits":"17","hi":"40652","lo":"-48960","policy":"derived","result":{"bits":"17","hi":"40652","lo":"-48960"},"tensor":"3"}],"saturated":false},{"acc32_params":[],"fits":["acc32","acc64"],"index":"3","intermediates":{},"npu_flags":[],"op":"relu","out":[{"bits":"17","hi":"40652","lo":"0","policy":"derived","result":{"bits":"17","hi":"40652","lo":"0"},"tensor":"4"}],"saturated":false},{"acc32_params":[],"fits":["acc32","acc64"],"index":"4","intermediates":{"convT.acc":{"bits":"29","hi":"149111536","lo":"-156916720"},"convT.term":{"bits":"23","hi":"3739984","lo":"-4187156"},"rq.p":{"bits":"29","hi":"149111536","lo":"-156916720"},"rq.q":{"bits":"21","hi":"582466","lo":"-612956"},"rq.t":{"bits":"29","hi":"149111536","lo":"-156916720"},"rq.y":{"bits":"21","hi":"582466","lo":"-612956"}},"k_c":"512","mac":{"a_bits":["17"],"w_bits":"8"},"npu_flags":[],"op":"conv_transpose","out":[{"bits":"21","hi":"582466","lo":"-612956","policy":"derived","result":{"bits":"21","hi":"582466","lo":"-612956"},"tensor":"5"}],"saturated":false},{"acc32_params":[],"fits":["acc32","acc64"],"index":"5","intermediates":{},"npu_flags":[],"op":"relu","out":[{"bits":"21","hi":"582466","lo":"0","policy":"derived","result":{"bits":"21","hi":"582466","lo":"0"},"tensor":"6"}],"saturated":false},{"acc32_params":[],"fits":["acc64"],"index":"6","intermediates":{"convT.acc":{"bits":"33","hi":"3092313901","lo":"-2428302245"},"convT.term":{"bits":"28","hi":"71643318","lo":"-64653726"},"rq.p":{"bits":"33","hi":"3092313901","lo":"-2428302245"},"rq.q":{"bits":"26","hi":"24158702","lo":"-18971112"},"rq.t":{"bits":"33","hi":"3092313901","lo":"-2428302245"},"rq.y":{"bits":"26","hi":"24158702","lo":"-18971112"}},"k_c":"29","mac":{"a_bits":["21"],"w_bits":"8"},"npu_flags":[],"op":"conv_transpose","out":[{"bits":"26","hi":"24158702","lo":"-18971112","policy":"derived","result":{"bits":"26","hi":"24158702","lo":"-18971112"},"tensor":"7"}],"saturated":false},{"acc32_params":[],"fits":["acc32","acc64"],"index":"7","intermediates":{},"npu_flags":[],"op":"clamp","out":[{"bits":"9","hi":"255","lo":"0","policy":"derived","result":{"bits":"9","hi":"255","lo":"0"},"tensor":"8"}],"saturated":false}],"scalars":[]}],"lowering":"int_synth/1","set_sha256":"7c15df3437b7f9a47aeec9107c782149e308b72f81c7a3981999769ab9eda1e5","verdict":{"acc32":false,"acc32_first_failure":{"bits":"33","graph":"synth","index":"6","key":"convT.acc","where":"node"},"acc64":true,"npu":false,"npu_flags":[],"npu_reasons":["acc32"]}}

View file

@ -1,63 +1,145 @@
"""Train a REAL int_synth decoder on REAL images and emit a committable
conformance fixture: the integer model (`int_synth.json`), an encoded test
image (`latent.bin`, the wire payload), and the integer-forward reference RGB
+ PSNR. The Rust decoder (`int_synth.rs`) reproduces the reference BIT-EXACTLY
on every machine — that is the whole point of int_synth: the determinism is
exact by construction, and this proves it on a real photograph, not a
constructed toy or a synthetic gradient.
#!/usr/bin/env python3
"""Train an int_synth decoder and emit the committable conformance vector `photo` of
`neural-conformance/`: the integer model (`int_synth.json`), a coded picture
(`latent.bin`, the wire payload), the integer-forward reference RGB (`ref_rgb.bin`) and
`meta.json`. The Rust decoder (`int_synth.rs`) reproduces the reference BIT-EXACTLY on
every machine: the determinism is exact by construction, and this vector shows it on a
model trained in float and quantised after training, not only on seeded weights.
Pipeline (the WORKING recipe — see wai_int_synth_train.py for why pure QAT
through the `>> shift` fails): float-train an autoencoder on random crops of
real photos, then post-training-quantize the DECODER with per-layer
activation-scale calibration so the integer pipeline tracks the float one.
Encode a held-out crop through the float encoder, quantize the latent to int8,
and decode it through the EXPORTED INTEGER model in numpy — the same i64
arithmetic int_synth.rs runs (scatter-accumulate, +bias, arithmetic `>> shift`,
integer ReLU, clamp [0,255]). That numpy decode is the reference the Rust
decoder must match to max|Δ|=0.
**The pictures are generated here; the tool reads no image file.** The pictures the
decoder is trained on, and the picture the vector codes, are written by `picture()`
below from a seed: 128 × 128 RGB8 dead-leaves pictures. A picture starts as a vertical
gradient between two colours; then 400 discs are painted back to front, each with a
centre uniform over the picture, a radius ⌊32·16 / (16 + u)⌋ (at least 2) for u uniform
in 0..239, a colour each channel of which is uniform in 16..239, and a linear shading
⌊(gx·dx + gy·dy) / 8⌋ for gx, gy uniform in −3..3, where (dx, dy) is the pixel's offset
from the centre; then every sample gets noise uniform in −2..2, and every value is
clamped to 0..255. Every draw is a SplitMix64 counter stream keyed by (seed, stream),
and the generator is integer arithmetic only, so a seed gives the same pictures on
every machine and every Python. The coded picture is stream 0 of the seed and the
training pictures are streams 1 to N: the decoder never sees the picture it codes.
python tools/wai_int_synth_real.py --src <dir-of-pngs> --out <fixture-dir>
Pipeline (the working recipe; see wai_int_synth_train.py for why training through the
`>> shift` fails): float-train an autoencoder on the training pictures, with torch
seeded from the same seed, deterministic algorithms and a fixed thread count on the
CPU: Adam steps on batches drawn by a generator seeded from the same seed, the learning
rate falling to a tenth for the last quarter of the steps, on the squared error of the
decoder's output before the clamp, so that an output below 0 or above 1 still has a
gradient; then quantise the DECODER after training, with each layer's activation scale
measured on the training pictures (its largest magnitude over 127, and 1/255 at the
output), its shift the largest that keeps its weights within ±127, and its weight scale
the one that makes the shift carry that activation scale exactly, so the integer
pipeline tracks the float one. Encode
the coded picture through the float encoder, quantise the latent to int8, and decode it
through the EXPORTED INTEGER model in numpy, the same i64 arithmetic int_synth.rs runs
(scatter-accumulate, +bias, arithmetic `>> shift`, integer ReLU, clamp to 0..255).
That numpy decode is the reference the Rust decoder must match to max|Δ|=0.
python tools/wai_int_synth_real.py --out <dir> [--seed S] [--pictures N] [--steps K]
python tools/wai_int_synth_real.py --picture-only <out.ppm> [--seed S] [--stream K]
python tools/wai_int_synth_real.py --check <vector-dir>
Training needs numpy and torch. `--picture-only` writes one generated picture as a
binary PPM, and `--check` regenerates the picture a committed vector codes from the
seed in its `meta.json`, requires the vector's reference decode (`ref_rgb.bin`) to
reconstruct it at the PSNR `meta.json` records, and requires the vector's record in the
corpus's `licences.json` to state the seed and streams `meta.json` records; both need
only the Python standard library. The committed vector's command is in
`neural-conformance/README.md`.
"""
from __future__ import annotations
import argparse
import glob
import hashlib
import json
import math
import struct
import sys
from pathlib import Path
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from PIL import Image
SIDE = 128 # the pictures' side; the model's output size
DISCS = 400 # discs per picture
THREADS = 4 # fixed: CPU reductions depend on it
SEED = 20261104 # the committed vector's seed
PICTURES = 512 # training pictures: streams 1 to PICTURES
STEPS, BATCH, LR = 3000, 32, 2e-4
# ---- the generated pictures (integer arithmetic only) --------------------------------
M64 = (1 << 64) - 1
GOLDEN = 0x9E3779B97F4A7C15
def load_crops(src: str, size: int, n: int, seed: int = 0, downscale: bool = False) -> torch.Tensor:
rng = np.random.default_rng(seed)
paths = sorted(glob.glob(str(Path(src) / "*.png")) + glob.glob(str(Path(src) / "*.jpg")))
if not paths:
raise SystemExit(f"no images in {src}")
if downscale:
# each image becomes a full recognizable scene at `size`×`size`.
out = [np.asarray(Image.open(p).convert("RGB").resize((size, size), Image.LANCZOS), np.float32) / 255.0
for p in paths]
return torch.from_numpy(np.stack([o.transpose(2, 0, 1) for o in out]))
crops = []
per = max(1, n // len(paths))
for p in paths:
im = np.asarray(Image.open(p).convert("RGB"), np.float32) / 255.0
H, W, _ = im.shape
for _ in range(per):
y = int(rng.integers(0, max(1, H - size)))
x = int(rng.integers(0, max(1, W - size)))
crops.append(im[y:y + size, x:x + size].transpose(2, 0, 1))
return torch.from_numpy(np.stack(crops[:n]))
def _mix64(x):
"""SplitMix64's finalizer."""
x &= M64
x = ((x ^ (x >> 30)) * 0xBF58476D1CE4E5B9) & M64
x = ((x ^ (x >> 27)) * 0x94D049BB133111EB) & M64
return x ^ (x >> 31)
class Rng:
"""A SplitMix64 stream, keyed by (seed, stream)."""
def __init__(self, seed, stream):
self.s = _mix64(seed * GOLDEN + stream * 0xD1B54A32D192ED03)
def next(self):
self.s = (self.s + GOLDEN) & M64
return _mix64(self.s)
def int(self, lo, hi):
"""Uniform integer in [lo, hi]."""
return lo + self.next() % (hi - lo + 1)
def picture(seed, stream, side=SIDE):
"""One dead-leaves picture as RGB8 bytes: rows top to bottom, pixels left to right,
R, G, B. The module docstring states the formula."""
r = Rng(seed, stream)
top = [r.int(16, 239) for _ in range(3)]
bottom = [r.int(16, 239) for _ in range(3)]
px = bytearray(side * side * 3)
for y in range(side):
row = [(top[c] * (side - 1 - y) + bottom[c] * y) // (side - 1) for c in range(3)]
for x in range(side):
p = (y * side + x) * 3
px[p:p + 3] = bytes(row)
rmax = side // 4
for _ in range(DISCS):
cx, cy = r.int(0, side - 1), r.int(0, side - 1)
rad = max(2, rmax * 16 // (16 + r.int(0, 239)))
base = [r.int(16, 239) for _ in range(3)]
gx, gy = r.int(-3, 3), r.int(-3, 3)
for y in range(max(0, cy - rad), min(side, cy + rad + 1)):
dy = y - cy
for x in range(max(0, cx - rad), min(side, cx + rad + 1)):
dx = x - cx
if dx * dx + dy * dy > rad * rad:
continue
shade = (gx * dx + gy * dy) // 8
p = (y * side + x) * 3
for c in range(3):
px[p + c] = min(255, max(0, base[c] + shade))
for i in range(len(px)):
px[i] = min(255, max(0, px[i] + r.int(-2, 2)))
return bytes(px)
def write_ppm(path, rgb, w, h):
Path(path).write_bytes(b"P6\n%d %d\n255\n" % (w, h) + bytes(rgb))
def psnr_db(a, b):
"""PSNR of two RGB8 buffers, in dB, from their integer sum of squared differences."""
assert len(a) == len(b) and len(a) > 0
sse = sum((x - y) * (x - y) for x, y in zip(a, b))
return 99.0 if sse == 0 else 10 * math.log10(255 * 255 * len(a) / sse)
# ---- the integer forward, byte-for-byte as int_synth.rs runs it ----
def conv_transpose2d_int(inp: np.ndarray, layer: dict) -> np.ndarray:
def conv_transpose2d_int(inp, layer):
import numpy as np
cin, hin, win = inp.shape
oc, k, s, p, op = layer["out_ch"], layer["k"], layer["stride"], layer["pad"], layer["out_pad"]
hout = (hin - 1) * s + k + op - 2 * p
@ -82,7 +164,9 @@ def conv_transpose2d_int(inp: np.ndarray, layer: dict) -> np.ndarray:
return out
def decode_int(latent_i8: np.ndarray, layers: list[dict], h: int, w: int) -> np.ndarray:
def decode_int(latent_i8, layers, h, w):
import numpy as np
cur = latent_i8.astype(np.int64)
for l in layers:
cur = conv_transpose2d_int(cur, l)
@ -90,38 +174,111 @@ def decode_int(latent_i8: np.ndarray, layers: list[dict], h: int, w: int) -> np.
return np.clip(cur, 0, 255).astype(np.uint8).transpose(1, 2, 0) # H,W,3
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--src", required=True, help="dir of real .png/.jpg photos")
ap.add_argument("--out", required=True, help="fixture output dir")
ap.add_argument("--chans", type=int, nargs="+", default=[64, 48, 32, 16, 3])
ap.add_argument("--latent-side", type=int, default=8)
ap.add_argument("--crops", type=int, default=160)
ap.add_argument("--epochs", type=int, default=2500)
ap.add_argument("--lr", type=float, default=2e-3)
ap.add_argument("--downscale", action="store_true", help="train on full images resized to the model size (recognizable whole scenes) instead of random crops")
ap.add_argument("--test-idx", type=int, default=0, help="which image (in --downscale mode) is the reconstructed fixture")
args = ap.parse_args()
torch.manual_seed(0)
dev = "mps" if torch.backends.mps.is_available() else "cpu"
GENERATOR = "tools/wai_int_synth_real.py" # this tool, as a licence record names it
def licence_record_fails(vdir, meta):
"""The vector's record in the corpus's `licences.json` (two directories up) states
the pictures meta.json says the vector was made from: content generated by this
tool, the coded picture `seed S, stream K`, and the training pictures
`seed S, streams A to B` from meta.json's `training_streams`, a range that leaves
out the coded picture's stream."""
fails = []
seed, stream = meta["seed"], meta["stream"]
tr_streams = meta.get("training_streams")
if not (isinstance(tr_streams, list) and len(tr_streams) == 2
and all(isinstance(x, int) for x in tr_streams) and tr_streams[0] <= tr_streams[1]):
return [f"meta.json's training_streams {tr_streams!r} is not [first, last]"]
a, b = tr_streams
if a <= stream <= b:
fails.append(f"meta.json's training streams {a} to {b} include the coded picture's stream {stream}")
try:
records = json.loads((vdir.parent.parent / "licences.json").read_text())["vectors"]
rec = records[vdir.name]
tr = rec["training"]
except (OSError, ValueError, KeyError, TypeError) as e:
return fails + [f"licences.json has no record with a training record for {vdir.name} ({e!r})"]
want = {
"": f"the dead-leaves picture the tool generates from seed {seed}, stream {stream}",
"training ": f"the dead-leaves pictures the tool generates from seed {seed}, streams {a} to {b}",
}
for label, r in (("", rec), ("training ", tr)):
if not isinstance(r, dict):
fails.append(f"licences.json's {label}record for {vdir.name} is not an object")
continue
if r.get("content") != "generated" or r.get("generator") != GENERATOR:
fails.append(f"licences.json's {label}record for {vdir.name} does not say its content is "
f"generated by {GENERATOR}")
if r.get("input") != want[label]:
fails.append(f"licences.json's {label}input for {vdir.name} is {r.get('input')!r}; "
f"meta.json says {want[label]!r}")
return fails
def check(vdir):
"""The committed vector codes the picture its seed generates: its reference decode
(`ref_rgb.bin`, whose SHA-256 is the corpus's digest for the vector in
`sha256.json` two directories up) reconstructs that picture at the PSNR meta.json
records, to two decimals; and the vector's licence record states the pictures
meta.json says it was made from (`licence_record_fails`)."""
vdir = Path(vdir)
meta = json.loads((vdir / "meta.json").read_text())
w, h = meta["width"], meta["height"]
head = (vdir / "latent.bin").read_bytes()[:12]
ph, pw, _c, _s, _r = struct.unpack("<IIHBB", head)
ref = (vdir / "ref_rgb.bin").read_bytes()
digests = json.loads((vdir.parent.parent / "sha256.json").read_text())
fails = []
if (w, h) != (SIDE, SIDE):
fails.append(f"meta.json says {w}x{h}, not the generator's {SIDE}x{SIDE}")
if (ph, pw) != (h, w):
fails.append(f"latent.bin's header says {pw}x{ph}, meta.json {w}x{h}")
if len(ref) != w * h * 3:
fails.append(f"ref_rgb.bin holds {len(ref)} bytes, not {w * h * 3}")
want = digests.get(vdir.name, {}).get("sha256")
if hashlib.sha256(ref).hexdigest() != want:
fails.append(f"ref_rgb.bin's SHA-256 is not sha256.json's {vdir.name} digest ({want})")
if not fails:
src = picture(meta["seed"], meta["stream"], w)
got = round(psnr_db(src, ref), 2)
print(f"{vdir}: seed {meta['seed']} stream {meta['stream']}: the reference decode reconstructs it at {got:.2f} dB")
if got != meta["psnr_db"]:
fails.append(f"PSNR {got:.2f} dB against the generated picture, meta.json records {meta['psnr_db']}")
fails += licence_record_fails(vdir, meta)
for f in fails:
print(f"FAIL {vdir}: {f}", file=sys.stderr)
return 1 if fails else 0
def train(args):
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(args.seed)
torch.use_deterministic_algorithms(True)
torch.set_num_threads(THREADS)
C, LS = args.chans[0], args.latent_side
n_layers = len(args.chans) - 1
size = LS * (2 ** n_layers)
print(f"device {dev} · image {size}×{size} · latent {LS}×{LS}×{C} · {n_layers} decoder layers")
assert size == SIDE, f"--latent-side and --chans give {size}x{size}; the pictures are {SIDE}x{SIDE}"
print(f"image {size}x{size} · latent {LS}x{LS}x{C} · {n_layers} decoder layers · seed {args.seed} · "
f"{args.pictures} training pictures · {args.steps} steps of {args.batch} · lr {args.lr} · {THREADS} threads")
if args.downscale:
data = load_crops(args.src, size, args.crops, downscale=True).to(dev)
test = data[args.test_idx:args.test_idx + 1] # one full scene the model represents
print(f"downscale mode: {data.shape[0]} full scenes; test image index {args.test_idx}")
else:
data = load_crops(args.src, size, args.crops).to(dev)
test = load_crops(args.src, size, 1, seed=999).to(dev) # held-out crop
def tensor(rgb):
a = np.frombuffer(rgb, np.uint8).reshape(size, size, 3).astype(np.float32) / 255.0
return a.transpose(2, 0, 1)
src = picture(args.seed, 0, size)
data = torch.from_numpy(np.stack([tensor(picture(args.seed, k, size)) for k in range(1, args.pictures + 1)]))
test = torch.from_numpy(tensor(src)[None])
dn = lambda a, b: nn.Conv2d(a, b, 5, 2, 2)
up = lambda a, b: nn.ConvTranspose2d(a, b, 5, 2, 2, output_padding=1)
enc = nn.Sequential(*sum([[dn(3 if i == 0 else C, C), nn.ReLU()] for i in range(n_layers)], [])[:-1]).to(dev)
dec = nn.ModuleList([up(args.chans[i], args.chans[i + 1]) for i in range(n_layers)]).to(dev)
enc = nn.Sequential(*sum([[dn(3 if i == 0 else C, C), nn.ReLU()] for i in range(n_layers)], [])[:-1])
dec = nn.ModuleList([up(args.chans[i], args.chans[i + 1]) for i in range(n_layers)])
def dfwd(y):
x = y
@ -132,43 +289,69 @@ def main() -> int:
return x
opt = torch.optim.Adam(list(enc.parameters()) + [p for c in dec for p in c.parameters()], lr=args.lr)
for ep in range(args.epochs):
# The learning rate falls to a tenth for the last quarter of the steps.
sched = torch.optim.lr_scheduler.MultiStepLR(opt, milestones=[args.steps * 3 // 4], gamma=0.1)
order = torch.Generator().manual_seed(args.seed)
for step in range(args.steps):
batch = data[torch.randperm(len(data), generator=order)[:args.batch]]
opt.zero_grad()
recon = dfwd(enc(data) * 64.0)
loss = F.mse_loss(recon.clamp(0, 1) * 255, data * 255)
loss.backward(); opt.step()
if ep % 250 == 0 or ep == args.epochs - 1:
print(f" epoch {ep:4d} train px-mse {loss.item():7.1f}")
recon = dfwd(enc(batch) * 64.0)
# Before the clamp: clamped, an output that starts below 0 has no gradient and
# the decoder can stay black.
loss = F.mse_loss(recon * 255, batch * 255)
loss.backward()
opt.step()
sched.step()
if step % 250 == 0 or step == args.steps - 1:
print(f" step {step:5d} batch px-mse {loss.item():7.1f}", flush=True)
# ---- calibrate activation scales on the training set, then quantize ----
def rgb8(t):
return (t[0].clamp(0, 1) * 255).round().to(torch.uint8).permute(1, 2, 0).contiguous().numpy().tobytes()
# ---- calibrate activation scales on the training set, then quantise ----
with torch.no_grad():
x = enc(data) * 64.0
amax = []
for i, c in enumerate(dec):
x = c(x)
if i < len(dec) - 1:
x = torch.relu(x)
amax.append(float(x.abs().max()))
s0 = float((enc(data) * 64.0).abs().max()) / 127.0
print(f" float decode of the coded picture: PSNR {psnr_db(src, rgb8(dfwd(enc(test) * 64.0))):.2f} dB")
s0max, amax = 0.0, [0.0] * n_layers
for k in range(0, len(data), args.batch):
x = enc(data[k:k + args.batch]) * 64.0
s0max = max(s0max, float(x.abs().max()))
for i, c in enumerate(dec):
x = c(x)
if i < len(dec) - 1:
x = torch.relu(x)
amax[i] = max(amax[i], float(x.abs().max()))
s0 = s0max / 127.0
scales = [s0] + [a / 127.0 for a in amax[:-1]] + [1.0 / 255.0]
layers, sin = [], s0
for i, c in enumerate(dec):
sw = float(c.weight.detach().abs().max()) / 127.0
shift = max(int(round(math.log2(scales[i + 1] / (sw * sin)))), 0)
# The layer's output is acc · sw · sin / 2^shift, so its scale is the calibrated
# one exactly when sw = sout / (sin · 2^shift). The shift is the largest that
# keeps every weight within ±127 under that sw. Rounding the shift alone would
# leave each layer's gain off by up to √2, and the errors would multiply down
# the stack into the output's brightness and contrast.
sout = scales[i + 1]
wmax = float(c.weight.detach().abs().max())
shift = math.floor(math.log2(sout * 127 / (wmax * sin)))
if shift < 0:
print(f" layer {i}: no shift >= 0 keeps the weights within ±127; clamping them")
shift = 0
sw = sout / (sin * 2 ** shift)
layers.append({
"in_ch": c.in_channels, "out_ch": c.out_channels, "k": 5, "stride": 2, "pad": 2, "out_pad": 1,
"weight": torch.clamp(torch.round(c.weight.detach() / sw), -128, 127).to(torch.int64).reshape(-1).tolist(),
"bias": torch.round(c.bias.detach() / (sw * sin)).to(torch.int64).reshape(-1).tolist(),
"shift": shift, "relu": i < len(dec) - 1,
})
sin = scales[i + 1]
sin = sout
# ---- encode the held-out test image → int8 latent → wire payload ----
# ---- encode the coded picture → int8 latent → wire payload ----
with torch.no_grad():
lat = (enc(test) * 64.0)[0].cpu().numpy() # (C, LS, LS)
lat = (enc(test) * 64.0)[0].numpy() # (C, LS, LS)
lat_i8 = np.clip(np.round(lat / s0), -128, 127).astype(np.int8)
out = Path(args.out); out.mkdir(parents=True, exist_ok=True)
out = Path(args.out)
out.mkdir(parents=True, exist_ok=True)
torch.save({"enc": enc.state_dict(), "dec": dec.state_dict()}, out / "float_model.pt")
(out / "int_synth.json").write_text(json.dumps({"layers": layers}, separators=(",", ":")))
# wire payload: u32 H, u32 W, u16 C, u8 S_log2, u8 reserved, then C*S*S int8
@ -177,28 +360,50 @@ def main() -> int:
(out / "latent.bin").write_bytes(payload)
# reference = the EXPORTED integer model decoded in numpy (== int_synth.rs)
recon = decode_int(lat_i8, layers, size, size)
orig = (test[0].cpu().numpy().transpose(1, 2, 0) * 255).round().astype(np.uint8)
Image.fromarray(recon).save(out / "recon.png")
Image.fromarray(orig).save(out / "original.png")
recon.tofile(out / "ref_rgb.bin")
recon = decode_int(lat_i8, layers, size, size).tobytes()
(out / "ref_rgb.bin").write_bytes(recon)
write_ppm(out / "original.ppm", src, size, size)
write_ppm(out / "recon.ppm", recon, size, size)
mse = float(np.mean((recon.astype(np.float64) - orig.astype(np.float64)) ** 2))
psnr = 99.0 if mse < 1e-9 else 10 * math.log10(255 * 255 / mse)
psnr = psnr_db(src, recon)
bpp = (len(payload) * 8) / (size * size)
# side-by-side (original | integer-decoded) for a demo, + metadata
Image.fromarray(np.concatenate([orig, recon], axis=1)).save(out / "compare.png")
(out / "meta.json").write_text(json.dumps({
"width": size, "height": size, "in_channels": C, "s_log2": s_log2,
"payload_bytes": len(payload), "psnr_db": round(psnr, 2), "bpp": round(bpp, 3),
}, indent=2))
print(f"\nwrote fixture → {out}")
print(f" model shifts {[l['shift'] for l in layers]} · latent {C}×{LS}×{LS} int8 ({len(payload)} B payload)")
print(f" real-image reconstruction: PSNR {psnr:.2f} dB · {bpp:.2f} bpp")
print(f" the integer decode is bit-identical on every machine; Rust int_synth must match ref_rgb.bin exactly")
"seed": args.seed, "stream": 0, "training_streams": [1, args.pictures],
"steps": args.steps, "batch": args.batch, "lr": args.lr, "threads": THREADS,
"torch": torch.__version__, "numpy": np.__version__,
}, indent=2) + "\n")
print(f"\nwrote the vector → {out}")
print(f" model shifts {[l['shift'] for l in layers]} · latent {C}x{LS}x{LS} int8 ({len(payload)} B payload)")
print(f" reconstruction of the coded picture: PSNR {psnr:.2f} dB · {bpp:.2f} bpp")
return 0
def main() -> int:
ap = argparse.ArgumentParser(description="Train an int_synth decoder on generated pictures and write the corpus vector.")
ap.add_argument("--out", help="output dir: int_synth.json, latent.bin, ref_rgb.bin and meta.json (the vector), "
"two previews and the float model (float_model.pt)")
ap.add_argument("--seed", type=int, default=SEED)
ap.add_argument("--pictures", type=int, default=PICTURES, help="training pictures: streams 1 to N of the seed")
ap.add_argument("--chans", type=int, nargs="+", default=[96, 72, 48, 24, 3])
ap.add_argument("--latent-side", type=int, default=8)
ap.add_argument("--steps", type=int, default=STEPS, help="Adam steps, each on a batch drawn from the training pictures")
ap.add_argument("--batch", type=int, default=BATCH)
ap.add_argument("--lr", type=float, default=LR)
ap.add_argument("--picture-only", metavar="OUT.ppm", help="write one generated picture and exit")
ap.add_argument("--stream", type=int, default=0, help="with --picture-only: 0 the coded picture, 1 to N the training pictures")
ap.add_argument("--check", metavar="VECTOR_DIR", help="check that a committed vector codes the picture its seed generates")
args = ap.parse_args()
if args.check:
return check(args.check)
if args.picture_only:
write_ppm(args.picture_only, picture(args.seed, args.stream), SIDE, SIDE)
return 0
if not args.out:
ap.error("--out is required unless --picture-only or --check is given")
return train(args)
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -36,7 +36,7 @@ pub fn committed() -> Vec<Set> {
members: vec![("int_synth.json", format!("neural-conformance/vectors/{n}/int_synth.json"))],
c: None,
note: if n == "photo" {
"neural-conformance; its decoder was trained on images the corpus does not record (see that corpus's provenance)"
"neural-conformance, generated: a decoder trained on generated pictures"
} else {
"neural-conformance, generated"
},

View file

@ -186,8 +186,9 @@ fn generate(dir: &Path) {
}),
);
}
// The committed REAL-photo vector: a model trained on real images
// (tools/wai_int_synth_real.py), not generated here. Decode it through the
// The committed trained vector: a model trained in float on pictures that
// tools/wai_int_synth_real.py generates from a seed, and quantised after
// training; that tool writes it, not this one. Decode it through the
// same integer path and CROSS-CHECK against the python integer-forward
// reference (ref_rgb.bin) — Rust must match it byte-for-byte (max|Δ|=0),
// proving the trained model decodes bit-identically across implementations.
@ -197,7 +198,7 @@ fn generate(dir: &Path) {
let img = decode(&pay, &photo.join("model")).expect("decode photo");
if let Ok(refrgb) = std::fs::read(photo.join("ref_rgb.bin")) {
assert_eq!(img.rgb, refrgb, "Rust int_synth diverges from the python integer-forward reference");
println!("trained real-photo vector: Rust == python integer-forward (max|Δ|=0)");
println!("trained vector photo: Rust == python integer-forward (max|Δ|=0)");
}
let model: serde_json::Value =
serde_json::from_slice(&std::fs::read(photo.join("int_synth.json")).unwrap()).unwrap();

View file

@ -15,6 +15,7 @@ import json
import os
import re
import socket
import socketserver
import sys
import threading
import time
@ -55,6 +56,14 @@ class Room(http.server.BaseHTTPRequestHandler):
pass
class Server(http.server.ThreadingHTTPServer):
"""HTTPServer looks up a name for its address before it listens, and nothing here reads the
name. On some hosts that lookup takes 35 s, past what `wai dev` waits for a service."""
def server_bind(self):
socketserver.TCPServer.server_bind(self)
def main(argv):
if argv[:1] in (["keygen"], ["relay-lines"]):
open(argv[1], "a").close()
@ -73,7 +82,7 @@ def main(argv):
while True:
c, _ = s.accept()
c.close()
http.server.ThreadingHTTPServer(listen_addr(config, "listen"), Room).serve_forever()
Server(listen_addr(config, "listen"), Room).serve_forever()
return 0

View file

@ -36,7 +36,9 @@ way down (the integer-entropy contract, SPEC §6), the reconstruction is
byte-identical on every machine, so the page re-checks its BLAKE3 against the
hash the native reference froze into `neural-conformance/expected_hashes.json`.
The four staged vectors (`demo/int_synth/`, copied from the corpus) include the
real *trained* `photo` model (128×128, 21.6 dB PSNR).
*trained* `photo` model: trained on dead-leaves pictures the corpus's tool
generates from a seed, it codes another picture of the same seed (128×128,
18.02 dB PSNR).
```
# from wai-web/: build the web pkg, then serve

View file

@ -37,7 +37,7 @@
</p>
<div class="bar">
<select id="vec">
<option value="photo">photo — real trained model, 128×128, 4 layers (PSNR 21.6 dB)</option>
<option value="photo">photo — trained model on a generated picture, 128×128, 4 layers (PSNR 18.0 dB)</option>
<option value="facade">facade — 72×72, 3 layers</option>
<option value="gradient">gradient — 38×38, 2 layers</option>
<option value="bloom">bloom — 18×18, 1 layer</option>

View file

@ -28,8 +28,8 @@
"width": 128,
"height": 128,
"layers": 4,
"rgb_hash": "1677d7112db6cadd70b0eeba8df263f9605cf3875809ce4b26e9c914f62fb582",
"psnr_db": 21.6,
"rgb_hash": "093d6c2d2670ecd23b2d338fcec2f49361098692a671429435f11ff50908c79d",
"psnr_db": 18.02,
"bpp": 3.006,
"in_channels": 96,
"s_log2": 3

File diff suppressed because one or more lines are too long