open-standards/wai/live-conformance/prior-forms

live-conformance/prior-forms — tensor files as prior forms

The corpus of extensions/prior-carriage.md §14: vectors.json and the files it names under files/.

vectors.json holds:

• natives: the native artifacts the corpus reads, each a file another corpus commits, named by its path from this directory and pinned by its SHA-256: the trained WSR1 priors generic and title and the seeded one seeded of ../upscale/priors/, and the generated WHM1 models hyper_g_s, hyper_h_s, world_t and world_g_s of ../../ops-conformance/sets/;
• files: every file of files/ with its SHA-256;
• vectors, each one of:
  • encode: a native artifact (native, or a file of files/natives/ in native_file) under a binding, written in a form (§10.3, §11). accept gives the file the writer must produce, byte for byte; refuse gives the code (§13);
  • load: a file in a form, loaded by §10.3. accept gives the binding and the native artifact the load must produce, byte for byte; refuse gives the code.

files/<prior>.<form> are the canonical files of each native artifact in each form (safetensors, gguf, nnc); files/natives/ holds native artifacts that do not read (trailing.wsr, badtag.whm) and one whose integers NNC's integer payload cannot carry (wide.whm); files/neg/ holds the negative controls, each a file refused on load with the code its vector states, so that the corpus gives every code of §13.

Exactness. A verifier conforms when it reaches every vector's outcome, its bytes and its code. A prior loaded from any of its three files is its native artifact, byte for byte.

Provenance. wai_prior_forms_vectors gen writes vectors.json and files/ from the seven native artifacts, which it reads and checks against their pins (prior_wire::corpus::INPUT_NATIVES). Every file is a function of those bytes: the encodes, and each negative control, made from the seeded prior's files by an edit written in code (prior_wire::corpus::generate). The two trained priors are the exact-upscale corpus's, trained there on generated pictures; no other data is in the corpus.

python3 -I ../prior_forms_verify.py .

Measured against another implementation (not in CI). prior-carriage §12.4 records a measurement of the NNC files against an open implementation of ISO/IEC 15938-17 that states compliance with its 2024 edition: its decoder reads them to the same integers, its encoder writes the same data units, and our reader reads its bitstreams. To repeat it, with that implementation's source at the commit §12.4 names (its licence is a BSD licence that grants no patent rights; its source is not in this repository):

python3 -m venv venv && venv/bin/pip install numpy==2.0.2 pybind11==2.13.6
cd <its source> && c++ -O2 -std=c++17 -shared -fPIC -undefined dynamic_lookup \
  $(../venv/bin/python -m pybind11 --includes) -Iextensions/deepCABAC/source \
  -Iextensions/deepCABAC/source/Lib -Iextensions/deepCABAC/source/Lib/CommonLib \
  extensions/deepCABAC/source/*.cpp extensions/deepCABAC/source/Lib/*/*.cpp \
  -o <build>/deepCABAC$(../venv/bin/python -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX"))')
PYTHONPATH=<build> venv/bin/python <wai>/tools/nnc_interop_check.py <its source> <wai>/live-conformance <wai>/live-conformance/prior-forms

It prints ALL_OK when every prior passes all three checks.