| Filename | Latest commit message | Latest commit date |
|---|---|---|
963410a8cc
The byte-exact JPEG-AI decoder in wai-rs is driven by weights lifted from the reference
ONNX assets, but the export step lived only in scratch and had to be rewritten each time
it was lost. This commits it.
Two regimes, and choosing wrongly yields plausible-but-wrong output:
ptq float nets (synthesis model_y/model_uv, hyper_decoder, MCM stages) -> 16-bit
per-tensor symmetric PTQ (conv -> int16 + mult=round(scale*2^31), shift=31;
bias/LayerNorm/temperature -> Q16), the layout Weights::from_json expects.
int hyper_scale_decoder -> verbatim. That net is integer end to end, so quantizing
it would corrupt an exactly-representable network; IntNet::from_json reads this.
Dependency-free: only the ONNX initializer table is needed, so it reads the protobuf
directly (no onnx/protobuf/torch import).
Verified to reproduce the weights the merged calibration tests were validated against:
synthesis_y 36 conv + 28 scalar (3,341,769 params, worst round-trip 6.6e-05), MCM stage0
3 conv (76,800 params, 6.9e-05), hyper_scale 9 integer tensors. Re-running
jpegai_hyperscale_calibration against freshly exported weights still reproduces the
reference scale indices 40960/40960 exactly.
Documents the eICCI caveat: post_filters/model_*.onnx export the model's `forward`,
which the decoder never calls, and their initializers are BN-folded and incomplete
(no BNY, scaleU/scaleV, or per-block scale) — take the reference state_dict instead.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
| .. | ||
| .github/workflows | ||
| audio-conformance | ||
| avatar-conformance | ||
| byte-exact-conformance | ||
| extensions | ||
| feed-conformance | ||
| film-conformance | ||
| glossary | ||
| haptic-conformance | ||
| neural-conformance | ||
| research | ||
| score-conformance | ||
| tools | ||
| vocab | ||
| volumetric-conformance | ||
| wai-relay | ||
| wai-rs | ||
| wai-spz-gen | ||
| wai-web | ||
| world-conformance | ||
| .gitignore | ||
| CHANGELOG.md | ||
| CONTRIBUTING.md | ||
| LICENSE | ||
| README.md | ||
| SPEC.md | ||
WAI — Web AI Media Transport & Execution
A container + capability-dispatch standard for media. WAI does not re-implement codecs; it dispatches to SOTA standard libraries (AVIF, JPEG-XL, PNG, JPEG, Opus, FLAC, AV1, zstd, XZ) and adds an envelope that lets the neural-shared-prior path coexist with the model-free floor.
Two paths open every conforming file:
- Neural condition — the sink regenerates the media against a
shared ambient prior (a model). The model is a requirement of
existence, never shipped (an
.mp3doesn't ship its decoder either). - Zeroth condition — a menu of registered SOTA codecs. Always satisfiable on any device that ships ffmpeg or equivalent.
The capability a file requires is named, not supplied.
What's here
| Path | Contents |
|---|---|
SPEC.md |
WAI v1.0 spec — envelope, manifest, capability menu |
wai-rs/ |
Rust reference implementation (lib + cdylib + staticlib + C FFI) |
glossary/ |
Reference catalog of media codecs surveyed for capability registration. codecs.json is the canonical 342-entry dataset; SCHEMA.md defines the shape and contribution rules. Rendered at wai.transaction.science/glossary. |
corpus/ |
Standard test material (Kodak/CLIC/Tecnick/SIPI images, Xiph Derf + UVG video, SQAM-class audio, enwik8 + Silesia text). Not committed; CORPUS.md documents reacquisition. |
Quickstart
cd wai-rs
cargo test --lib # every registered capability + envelope round-trip
cargo build --release --bin wai
cargo build --release --lib # produces libwai.dylib/.so/.dll + libwai.a
On macOS, libflac lives under the Homebrew "flac" formula prefix that
Cargo doesn't probe automatically. The committed .cargo/config.toml
adds the path; if you build outside the repo set
RUSTFLAGS="-L /opt/homebrew/opt/flac/lib".
SDK consumption
wai-rs exposes a stable C ABI in src/ffi.rs —
the cdylib drops in as libwai.dylib/.so/.dll. Bindings for any
language (Python ctypes, Node N-API, Swift, JNI, Go cgo, etc.) call
into:
wai_image_{png,jpeg,avif,jxl}_{encode,decode}wai_audio_{opus,flac}_{encode,decode}wai_text_{zstd,xz}_{encode,decode}wai_envelope_{pack,unpack}— wrap/unwrap the WAI containerwai_buffer_free,wai_last_error— memory + error handling
Registered capabilities (the zeroth menu)
| Capability | Codec | Lossless? | Mandatory? |
|---|---|---|---|
wai.image.png |
PNG | yes | mandatory floor |
wai.image.jxl |
JPEG-XL | both | recommended |
wai.image.avif |
AVIF (AV1-based) | lossy | recommended |
wai.image.jpeg |
JPEG | lossy | recommended (legacy compat) |
wai.audio.flac |
FLAC | yes | mandatory floor |
wai.audio.opus |
Opus | lossy | recommended |
wai.video.av1 |
AV1 | lossy | recommended |
wai.video.av1.lossless |
AV1 (quantizer=0) |
YUV-lossless | recommended |
wai.text.zstd |
zstd | yes | mandatory floor |
wai.text.xz |
XZ/LZMA2 | yes | recommended |
Neural capabilities (wai.neural.*) are OPTIONAL and sink-advertised. Most
run float inference, so their decode is exact only on the build that produced
it. A distinguished subset is integer-exact by construction — an all-i64
decode with no float and no ONNX — so the reconstructed artifact is
byte-identical on every machine and its hash is portable. Conformance is a
hash: a deployer can require intent=replicate to dispatch only to these.
| Capability | Integer-exact decode of | Conformance |
|---|---|---|
wai.neural.int_hyper |
image — integer hyperprior σ source + integer entropy + integer synthesis | byte-exact (hash-portable) |
wai.neural.int_synth |
image — integer synthesis only | byte-exact (hash-portable) |
wai.video.int_hyper |
video — int_hyper I-frame + integer half-pel motion-compensated P-frames |
byte-exact (hash-portable) |
wai.audio.int_codec |
audio — learned 1-D transform, integer synthesis → int16 PCM | byte-exact (hash-portable) |
wai.splat.int_codec |
3D — learned Gaussian-splat-attribute transform, integer synthesis | byte-exact (hash-portable) |
Interactive & temporal content (extensions)
Beyond the codec menu, WAI carries content that consumes input or moves
over time — reconstructed from instructions at the sink, over one
wai.det.fixed64 fixed-point floor, each with its own deterministic
*-equivalence conformance criterion:
| Capability | What it carries | Conformance |
|---|---|---|
wai.world.{scene,replay,live} |
interactive worlds (signed op log) | replay-equivalence |
wai.world.model |
action-conditioned world model (signed action ops → next frame, integer transition + synthesis) | replay-equivalence |
wai.mesh.glb / wai.splat.spz |
3D mesh / Gaussian-splat assets | content-addressed |
wai.audio.scene |
object-based spatial audio | mixdown-equivalence |
wai.audio.score |
music as note events (synthesized at the sink) | mixdown-equivalence |
wai.haptic.signal |
keyframed touch envelopes | sample-equivalence |
wai.feed.state |
scoreboards / telemetry / market feeds | replay-equivalence |
wai.avatar.pose |
digital humans (skeletal pose stream) | pose-equivalence |
wai.splat.4d |
volumetric / 4D temporal Gaussian splats | frame-state-equivalence |
wai.film.linear |
animation / CGI film (sealed world + camera track) | film-equivalence |
wai.film.reel |
a film with cuts (one world, a sequence of shots) | reel-equivalence |
And the verify half — a signed, energy-accounted lineage for opaque
captured content WAI does not reconstruct (wai.asset.provenance).
See SPEC.md §5 and extensions/ for each.
License
Apache-2.0 — see LICENSE.