d7037c7c7b
prior_forms_verify.py implements prior-carriage §9 to §12 and Appendix A from the text (strict readers, canonical writers, the arithmetic coder both ways, both bindings) and agrees with wai-rs on every vector; it also flips bytes of every accepted file and requires each copy to be caught. §12.4 records the measurement: an open implementation of ISO/IEC 15938-17 reads our NNC files to the same integers, its encoder writes byte-identical data units, and our reader reads its bitstreams. tools/nnc_interop_check.py is the check; it runs that implementation's code and is not a CI step. Co-Authored-By: Claude Opus 5.5 <[email protected]>
60 lines
3.5 KiB
Markdown
60 lines
3.5 KiB
Markdown
# live-conformance/prior-forms — tensor files as prior forms
|
|
|
|
The corpus of [`extensions/prior-carriage.md`](../../extensions/prior-carriage.md) §14:
|
|
`vectors.json` and the files it names under `files/`.
|
|
|
|
`vectors.json` holds:
|
|
|
|
- `natives`: the native artifacts the corpus reads, each a file another corpus commits,
|
|
named by its path from this directory and pinned by its SHA-256: the trained `WSR1`
|
|
priors `generic` and `title` and the seeded one `seeded` of
|
|
[`../upscale/priors/`](../upscale/priors/), and the generated `WHM1` models `hyper_g_s`,
|
|
`hyper_h_s`, `world_t` and `world_g_s` of
|
|
[`../../ops-conformance/sets/`](../../ops-conformance/sets/);
|
|
- `files`: every file of `files/` with its SHA-256;
|
|
- `vectors`, each one of:
|
|
- `encode`: a native artifact (`native`, or a file of `files/natives/` in `native_file`)
|
|
under a `binding`, written in a `form` (§10.3, §11). `accept` gives the `file` the
|
|
writer must produce, byte for byte; `refuse` gives the `code` (§13);
|
|
- `load`: a `file` in a `form`, loaded by §10.3. `accept` gives the `binding` and the
|
|
`native` artifact the load must produce, byte for byte; `refuse` gives the `code`.
|
|
|
|
`files/<prior>.<form>` are the canonical files of each native artifact in each form
|
|
(`safetensors`, `gguf`, `nnc`); `files/natives/` holds native artifacts that do not read
|
|
(`trailing.wsr`, `badtag.whm`) and one whose integers NNC's integer payload cannot carry
|
|
(`wide.whm`); `files/neg/` holds the negative controls, each a file refused on load with
|
|
the code its vector states, so that the corpus gives every code of §13.
|
|
|
|
**Exactness.** A verifier conforms when it reaches every vector's outcome, its bytes and
|
|
its code. A prior loaded from any of its three files is its native artifact, byte for byte.
|
|
|
|
**Provenance.** `wai_prior_forms_vectors gen` writes `vectors.json` and `files/` from the
|
|
seven native artifacts, which it reads and checks against their pins
|
|
(`prior_wire::corpus::INPUT_NATIVES`). Every file is a function of those bytes: the
|
|
encodes, and each negative control, made from the seeded prior's files by an edit
|
|
written in code (`prior_wire::corpus::generate`). The two trained priors are the
|
|
exact-upscale corpus's, trained there on generated pictures; no other data is in the
|
|
corpus.
|
|
|
|
```
|
|
python3 -I ../prior_forms_verify.py .
|
|
```
|
|
|
|
**Measured against another implementation (not in CI).** prior-carriage §12.4 records a
|
|
measurement of the NNC files against an open implementation of ISO/IEC 15938-17 that
|
|
states compliance with its 2024 edition: its decoder reads them to the same integers, its
|
|
encoder writes the same data units, and our reader reads its bitstreams. To repeat it,
|
|
with that implementation's source at the commit §12.4 names (its licence is a BSD licence
|
|
that grants no patent rights; its source is not in this repository):
|
|
|
|
```
|
|
python3 -m venv venv && venv/bin/pip install numpy==2.0.2 pybind11==2.13.6
|
|
cd <its source> && c++ -O2 -std=c++17 -shared -fPIC -undefined dynamic_lookup \
|
|
$(../venv/bin/python -m pybind11 --includes) -Iextensions/deepCABAC/source \
|
|
-Iextensions/deepCABAC/source/Lib -Iextensions/deepCABAC/source/Lib/CommonLib \
|
|
extensions/deepCABAC/source/*.cpp extensions/deepCABAC/source/Lib/*/*.cpp \
|
|
-o <build>/deepCABAC$(../venv/bin/python -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX"))')
|
|
PYTHONPATH=<build> venv/bin/python <wai>/tools/nnc_interop_check.py <its source> <wai>/live-conformance <wai>/live-conformance/prior-forms
|
|
```
|
|
|
|
It prints `ALL_OK` when every prior passes all three checks.
|