wai: neural-conformance's trained vector is trained and coded on generated pictures, and every vector has a licence record #639

Merged

dcharlot merged 3 commits from wai/neural-corpus-licence into main 2026-11-06 14:18:19 -05:00

Conversation 0 Commits 3 Files changed 29 +884 -138

dcharlot commented 2026-11-05 04:26:16 -05:00

Owner

Copy link

What this is

neural-conformance's trained vector, photo, coded a third-party picture with no recorded licence for redistribution, and its decoder was trained on pictures the corpus did not record. Now its decoder is trained on pictures a tool of this repository generates from a seed, and it codes one of them. Every vector of the corpus now has a licence record, and a repository check requires one.

Merge order. This branch sits on #632's commit, 946f3beda, which moves changelog entries into wai/changes/ and the conformance workflow's lists into step files. Merge #632 first; until it merges, this pull request's diff also shows #632's change.

What it carries. One commit, ae94f935e: the change of wai/neural-corpus-licence (44d5f2264..6bddde42d) in #632's layout. Its message lists the source commits. The branch's changelog entry is a file of wai/changes/. Its edits to wai-conformance.yml are not lists that #632 derives, so they stay in the workflow: the wai/tools/wai_int_synth_real.py trigger path, and two steps of the cargo-classes job. Those two steps now follow the audio provenance step, instead of the pin step, which wai/conform-output removes. Its edits to repo-checks.yml and wai/CONTRIBUTING.md sit beside #632's own edits to those files, as the source branch has them. Every other file is as the source branch has it.

What it changes

photo is generated content. Its decoder is trained on dead-leaves pictures, and its latent codes one, all generated by tools/wai_int_synth_real.py from a seed. The vector keeps its name, its shape, its payload format and its capability. Its weights, latent and digests are new.

The tool reads no image file.

• picture() writes each picture from a seed in integer arithmetic only, with a SplitMix64 stream per picture, so a seed gives the same pictures on every machine and every Python.
• The coded picture is stream 0 of the vector's seed, and the training pictures are streams 1 to 512, so the decoder never sees the picture it codes.
• Training runs on the CPU, seeded from the same seed, with deterministic algorithms, a fixed thread count and seeded batches.
• The loss is taken before the output's clamp. Each layer's weight scale is chosen so that its shift carries its calibrated activation scale exactly.
• --picture-only writes one picture.
• --check regenerates the picture a vector codes from the seed in its meta.json. It requires the vector's reference decode, whose SHA-256 must be the corpus's digest for the vector, to reconstruct that picture at the PSNR meta.json records. It also requires the vector's licence record to state the seed, the coded picture's stream and the training streams that meta.json records, and requires those training streams to leave out the coded picture's.
• Both --picture-only and --check need only the Python standard library.

Regenerated through their generators.

• vectors/photo/ is regenerated by the tool.
• expected_hashes.json and sha256.json are regenerated by wai_int_synth_vectors gen, which checks the Rust decode against the tool's integer forward pass and leaves the other vectors unchanged.
• The photo set's certificate and lowered graph in ops-conformance/certificates/, and index.json, are regenerated by wai-verify vectors gen.
• The demo's byte copies in wai-web/demo/int_synth/ are updated, and the demo page and its README give the vector's PSNR and say its picture is generated.

The set still meets acc64 and fails acc32 at node 6, convT.acc. wai-ops.yml names the new certificate and the new accumulator of the node's witness.

Licence records. neural-conformance/licences.json (wai-corpus-licences/1) records, for each vector:

• its licence, as an SPDX identifier;
• whether its content is generated here or is a third-party input;
• the generator and its input, or the source;
• for a trained vector, the same of its training data.

Every vector is Apache-2.0. wai/CONTRIBUTING.md (Corpus content) states the record.

Records kept. The sink-energy tables measured on the earlier photo name its latent by its SHA-256 in their procedure.txt; they are records of those runs and are unchanged. int_video's video_real is in another corpus and is unchanged.

What the checks hold

• scripts/check-wai-corpus-licences.py is a new step of the repository checks, run on every change. It fails on each of these:
  • a vector without a record (a directory under vectors/, or a name in either hash manifest);
  • a record for no vector;
  • a licence outside its list of licences that permit redistribution;
  • a record that leaves out what its content needs;
  • a trained vector without a training record;
  • an unknown member.
• scripts/check-wai-corpus-licences-selftest.sh, the step before it, builds a scratch corpus and shows that the check fails each of those ways.
• The picture check. One leg of the conformance workflow's cargo-classes job runs wai_int_synth_real.py --check on photo; the generator is integer arithmetic, so one leg is enough. The step after it runs the same check on a copy whose licence record gives the training pictures as streams 1 to 64, and requires the check to fail and say why.
• The corpus. The int-synth corpus step of cargo-classes runs wai_int_synth_vectors over neural-conformance/, and wai-ops.yml's bounds checks hold the photo set to its new certificate and witness.

How it was verified

This commit, ae94f935e.

• Locally (git and Python checks only): scripts/wai-conformance-lists.py selftest and check, scripts/wai-conformance-lists-selftest.sh, scripts/check-changelog-fragment.py against main, scripts/changelog-assemble.py --check, scripts/changelog-fragments-selftest.sh, scripts/check-wai-corpus-licences.py and its self-test all pass. Rebasing from the conversion's first base, #632's earlier tip 14f1abb7c, onto 946f3beda left every file's change the same.
• On a Linux x86_64 build host: in wai/wai-rs, cargo check --locked --no-default-features --features neural_int --tests; and in wai/wai-ops, cargo check --locked --tests. Both pass.
• With the other converted branches. This branch merges without conflict with each of the other branches converted to #632's layout (by git merge-tree). Merging all of them in forward, reverse and a mixed order gives one tree. On the build host, that tree passes these runs: cargo check --tests under the derived WAI_FEATURES and under WAI_PQ_FEATURES; cargo check --tests of quickstart,app; the catalogue's library tests; and the wai binary's unit tests.

The picture check, wai_int_synth_vectors' verification and wai-ops.yml's bounds checks did not run on this commit here; their results come from this pull request's CI.

On the CI hosts. Results on the CI runners come from this pull request's own CI run. None of the runs above is a CI result.

🤖 Generated with Claude Code

## What this is neural-conformance's trained vector, `photo`, coded a third-party picture with no recorded licence for redistribution, and its decoder was trained on pictures the corpus did not record. Now its decoder is trained on pictures a tool of this repository generates from a seed, and it codes one of them. Every vector of the corpus now has a licence record, and a repository check requires one. **Merge order.** This branch sits on #632's commit, 946f3beda, which moves changelog entries into `wai/changes/` and the conformance workflow's lists into step files. Merge #632 first; until it merges, this pull request's diff also shows #632's change. **What it carries.** One commit, ae94f935e: the change of `wai/neural-corpus-licence` (44d5f2264..6bddde42d) in #632's layout. Its message lists the source commits. The branch's changelog entry is a file of `wai/changes/`. Its edits to `wai-conformance.yml` are not lists that #632 derives, so they stay in the workflow: the `wai/tools/wai_int_synth_real.py` trigger path, and two steps of the cargo-classes job. Those two steps now follow the audio provenance step, instead of the pin step, which `wai/conform-output` removes. Its edits to `repo-checks.yml` and `wai/CONTRIBUTING.md` sit beside #632's own edits to those files, as the source branch has them. Every other file is as the source branch has it. ## What it changes **`photo` is generated content.** Its decoder is trained on dead-leaves pictures, and its latent codes one, all generated by `tools/wai_int_synth_real.py` from a seed. The vector keeps its name, its shape, its payload format and its capability. Its weights, latent and digests are new. **The tool reads no image file.** - `picture()` writes each picture from a seed in integer arithmetic only, with a SplitMix64 stream per picture, so a seed gives the same pictures on every machine and every Python. - The coded picture is stream 0 of the vector's seed, and the training pictures are streams 1 to 512, so the decoder never sees the picture it codes. - Training runs on the CPU, seeded from the same seed, with deterministic algorithms, a fixed thread count and seeded batches. - The loss is taken before the output's clamp. Each layer's weight scale is chosen so that its shift carries its calibrated activation scale exactly. - `--picture-only` writes one picture. - `--check` regenerates the picture a vector codes from the seed in its `meta.json`. It requires the vector's reference decode, whose SHA-256 must be the corpus's digest for the vector, to reconstruct that picture at the PSNR `meta.json` records. It also requires the vector's licence record to state the seed, the coded picture's stream and the training streams that `meta.json` records, and requires those training streams to leave out the coded picture's. - Both `--picture-only` and `--check` need only the Python standard library. **Regenerated through their generators.** - `vectors/photo/` is regenerated by the tool. - `expected_hashes.json` and `sha256.json` are regenerated by `wai_int_synth_vectors gen`, which checks the Rust decode against the tool's integer forward pass and leaves the other vectors unchanged. - The photo set's certificate and lowered graph in `ops-conformance/certificates/`, and `index.json`, are regenerated by `wai-verify vectors gen`. - The demo's byte copies in `wai-web/demo/int_synth/` are updated, and the demo page and its README give the vector's PSNR and say its picture is generated. The set still meets `acc64` and fails `acc32` at node 6, `convT.acc`. `wai-ops.yml` names the new certificate and the new accumulator of the node's witness. **Licence records.** `neural-conformance/licences.json` (`wai-corpus-licences/1`) records, for each vector: - its licence, as an SPDX identifier; - whether its content is generated here or is a third-party input; - the generator and its input, or the source; - for a trained vector, the same of its training data. Every vector is Apache-2.0. `wai/CONTRIBUTING.md` (*Corpus content*) states the record. **Records kept.** The sink-energy tables measured on the earlier `photo` name its latent by its SHA-256 in their `procedure.txt`; they are records of those runs and are unchanged. `int_video`'s `video_real` is in another corpus and is unchanged. ## What the checks hold - **`scripts/check-wai-corpus-licences.py`** is a new step of the repository checks, run on every change. It fails on each of these: - a vector without a record (a directory under `vectors/`, or a name in either hash manifest); - a record for no vector; - a licence outside its list of licences that permit redistribution; - a record that leaves out what its content needs; - a trained vector without a training record; - an unknown member. - **`scripts/check-wai-corpus-licences-selftest.sh`**, the step before it, builds a scratch corpus and shows that the check fails each of those ways. - **The picture check.** One leg of the conformance workflow's cargo-classes job runs `wai_int_synth_real.py --check` on `photo`; the generator is integer arithmetic, so one leg is enough. The step after it runs the same check on a copy whose licence record gives the training pictures as streams 1 to 64, and requires the check to fail and say why. - **The corpus.** The int-synth corpus step of cargo-classes runs `wai_int_synth_vectors` over `neural-conformance/`, and `wai-ops.yml`'s bounds checks hold the photo set to its new certificate and witness. ## How it was verified **This commit, ae94f935e.** - **Locally** (git and Python checks only): `scripts/wai-conformance-lists.py selftest` and `check`, `scripts/wai-conformance-lists-selftest.sh`, `scripts/check-changelog-fragment.py` against main, `scripts/changelog-assemble.py --check`, `scripts/changelog-fragments-selftest.sh`, `scripts/check-wai-corpus-licences.py` and its self-test all pass. Rebasing from the conversion's first base, #632's earlier tip 14f1abb7c, onto 946f3beda left every file's change the same. - **On a Linux x86_64 build host:** in `wai/wai-rs`, `cargo check --locked --no-default-features --features neural_int --tests`; and in `wai/wai-ops`, `cargo check --locked --tests`. Both pass. - **With the other converted branches.** This branch merges without conflict with each of the other branches converted to #632's layout (by `git merge-tree`). Merging all of them in forward, reverse and a mixed order gives one tree. On the build host, that tree passes these runs: `cargo check --tests` under the derived `WAI_FEATURES` and under `WAI_PQ_FEATURES`; `cargo check --tests` of `quickstart,app`; the catalogue's library tests; and the `wai` binary's unit tests. The picture check, `wai_int_synth_vectors`' verification and `wai-ops.yml`'s bounds checks did not run on this commit here; their results come from this pull request's CI. **On the CI hosts.** Results on the CI runners come from this pull request's own CI run. None of the runs above is a CI result. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 1 commit 2026-11-05 04:26:17 -05:00

wai/neural-corpus-licence, converted to the layout of the fix: no shared append point ... ae94f935e7

The change of wai/neural-corpus-licence (44d5f2264..6bddde42d, 5 commits), on the fix branch: its
changelog entries as files of wai/changes/, its wai-conformance.yml edits as step files,
corpus files and test tags, and every other file as the branch has it.

- 40052a902 wai: the trained int_synth vector's tool generates its own pictures from a seed
- c86d58210 wai: neural-conformance's photo, trained and coded on generated pictures
- 4321d68cc wai: every neural-conformance vector has a licence record, and a lint requires one
- 63aea099a wai: photo's licence record gives its training pictures as streams 1 to 512
- 6bddde42d wai: the trained int_synth vector's check holds its licence record to meta.json

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 1 commit 2026-11-05 10:07:10 -05:00

neural-conformance: the entry acknowledges the nine intended breaks of the retrained vector ... 164c842056

wai breaking reports the retrained photo vector's files and the operator certificate that
certifies its decoder as breaking: intended, since the vector is retrained and recoded on
generated pictures. Each is acknowledged in this change's entry.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 1 commit 2026-11-06 02:20:58 -05:00

wai dev's test: the stand-in room service listens without a reverse lookup ... 2f274c3939

tests/cli_studio.rs failed on every run on one macOS runner and passed on
every other runner, 3 of 3 against 19 of 19 on 5 November, with "the room
service's clock on the relay: not within 30 s".

The stand-in room service starts an HTTPServer, whose server_bind calls
socket.getfqdn before the server listens. On that host, under the runner's
Python 3.15.2, getfqdn("127.0.0.1") takes 35 s, past the 30 s wai dev waits, and
a stack dump of the stalled stand-in shows it there. The stand-in relay opens a
plain socket and answers in 0.1 s. Nothing here reads the name, so the fixture
no longer asks for it.

On that host and Python the room stand-in took 35.5 s and now takes 0.12 s. A
socket.getfqdn planted to stall for 40 s fails the test without this change,
with the same message, and passes with it.

Changelog: none

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot merged commit d49a0c27aa into main 2026-11-06 14:18:19 -05:00

dcharlot referenced this pull request from a commit 2026-11-06 14:18:20 -05:00

Merge pull request 'wai: neural-conformance's trained vector is trained and coded on generated pictures, and every vector has a licence record' (#639) from wai/neural-corpus-licence into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!639

No description provided.