open-standards/wai/SOTA-GAP-2026.md

WAI — SOTA gap analysis & roadmap (2026)

What WAI needs to sit at the 2025–2026 frontier of media transport + neural/semantic coding, and the concrete work to get there. Derived from a multi-source research pass (24 sources, 111 extracted claims). Confidence: ✅ = adversarially verified (3-of-3); ◐ = single credible primary source (the verify pass was rate-limited, so most ◐ items are unverified-but-sourced and worth a clean re-check).

STATUS 2026-06-09 — roadmap essentially CLOSED

Everything below that could be implemented has been, in wai-rs, each decoder validated end-to-end against a real or faithful reference:

• Learned image, full conditional-entropy family: bmshj2018 (factorized) → mbt2018_mean (hyperprior) → elic (checkerboard) → mlicpp (channel-autoregressive) → mlicv2 / cmic (SOTA, live routes via the mlicpp core) → glc (categorical/VQ — softmax → CompressAI-bit-exact pmf_to_quantized_cdf → categorical rANS). bmshj2018 + mbt2018_mean are checked byte-for-byte against pretrained CompressAI models.
• Inter-frame video: dcvc_rt/dcvc_fm decode orchestration (neural/video_dcvc.rs) — frame loop + reference-feature propagation on CPU via ort. "Requires CUDA" was a speed claim, not a decode requirement.
• Audio: snac multi-resolution decoder validated against the real snac_24khz (after export surgery: zero-noise → deterministic + reshape-free snake; tools/wai_snac_export.py).
• Integer-exact replicate member is REAL: a trained, activation-calibrated integer synthesis (tools/wai_int_synth_train.py) decodes bit-identically (max|Δ|=0) in Rust. (Also fixed a zero-gradient torch.floor bug in the QAT trainer.)
• Reproducible validation: tools/wai_validate_synthetic.py regenerates every synthetic harness; the gated tests in neural_native_decode.rs consume them.

The JPEG AI "access boundary" is RESOLVED (2026-06-10). The reference software is public (BSD-3): gitlab.com/wg1/jpeg-ai/jpeg-ai-reference-software (Rec. ITU-T T.840.1 | ISO/IEC 6048-1, encoder+decoder, pretrained VM models incl. integer VM_common_int, weights via git-lfs/DVC). And the port has begun with the load-bearing piece: the mANS entropy engine now decodes in Rust (wai-rs/src/neural/jpegai_mans.rs) — backward-read dual-state tANS, decode_sgm (σ-bucket residuals incl. the literal escape) + decode_factorize (per-channel byte-CDF hyper-latents incl. escape-63), plus an integer-only port of the decode-table builder. Golden-validated against bitstreams produced by the reference software's own encoder (built locally with an omp shim): table builder bit-exact, both stream types decode exactly (tests/jpegai_mans_golden.rs; fixture harness tools/wai_jpegai_goldens.py). Pure integer — no float, no ONNX — so it joins the byte-exact family.

Genuinely still open:

1. JPEG AI full decode pipeline — ordinary porting work now, not access: bitstream-structure parsing + the synthesis networks (the reference repo ships models_export/ and the VM weights; the normative σ-PMF constants are literals in ec_lib_mans.py). The entropy engine — the part that must be bit-exact — is done and validated.

Cross-hardware CI byte-equality matrix — SET UP (2026-06-10). .github/workflows/wai-byte-equality.yml runs wai/byte-exact-conformance on five OS/arch legs (linux x86_64 + aarch64, macOS arm64 + x86_64, windows x86_64). The conformance crate #[path]-includes the pure-integer sources from wai-rs (rANS + int_synth — no system codecs, no ONNX runtime → ~1-minute legs) and asserts goldens: two rANS streams produced by CompressAI's upstream encoder (Gaussian-table path incl. the bypass escape, and the categorical path) plus an int_synth output SHA-256 from the reference machine (macOS arm64). One diverging byte on any platform fails the suite. Activates on the next push of the workflow + crate.

Sections below are kept as the original analysis + per-item history.

WAI's three paths, for reference:

1. Floor + modern menu — PNG/FLAC/zstd floor; AVIF/JPEG-XL/JPEG/Opus/AV1/XZ modern; bit-exact decode-equivalence.
2. Neural / bit-exact (replicate) — ship compact conditioning + integer rANS; regenerate at the sink against a named, hash-pinned prior with a determinism contract.
3. Perceptual (wai.semantic.*) — ship a semantic package (CLIP embed + thumb + edges); regenerate with any conforming generator; conform by similarity threshold, not bytes.

---

The load-bearing finding: the determinism contract was too weak

Float round-off is platform-dependent, and in a learned entropy coder a single diverging bit corrupts the whole decode. So a single-threaded-float pin is reproducible on the same build but can still diverge across hardware — it cannot honestly back a normative replicate. Only integer-only inference is byte-identical everywhere.

• ◐ Integer networks fix cross-platform encode/decode of latent-variable models — float round-off is the named mechanism. — Google Research
• ✅ Post-training quantization to integer-arithmetic-only gives deterministic cross-platform inference without retraining, preserving rate-distortion (re-fetched + confirmed). — arXiv 2202.07513
• ◐ A single bit difference in a learned entropy coder can crash the decoder across hardware. — arXiv 2410.09872
• ✅ DCVC-RT ships this in practice: "model integerization for consistent cross-device coding." — arXiv 2502.20762

✅ P0 — DONE (in code): integer-exact is the bar for replicate

Implemented in wai-rs/src/container.rs:

• DeterminismClass { IntegerExact, FloatReproducible, Unknown } + Prior::determinism_class() / Prior::is_cross_hardware_exact(). Any float token (f32/f64/float) downgrades to FloatReproducible even if the entropy stage is integer — so "int-rans/p16+onnx-cpu-f32/1thread" (bmshj2018: integer rANS, float synthesis) honestly classifies as FloatReproducible, i.e. exact here, not exact everywhere.
• replicate_is_sound(intent, prior) — a replicate claim is sound only with an integer-exact prior; float-reproducible decodes should be tagged perceptual / improve.
• Unit-tested.

P0 conformance gate — DONE. Wai::replicate_is_sound() + neural::decode_envelope_strict() reject a replicate envelope whose prior isn't integer-exact (DecodeError::UnsoundReplicate), before touching the model. decode_envelope stays lenient (decodes anyway, honest about "exact on this build"). Model-independent gate test green.

P0 integer-exact decode — IMPLEMENTED. wai.neural.int_synth (wai-rs/src/neural/int_synth.rs) is a synthesis whose reconstruction is bit-identical on every machine by construction: an all-i64 transposed-conv stack with integer requantize + integer ReLU — no float, no ONNX, nothing implementation-defined. This is the first capability that can soundly back intent=replicate cross-hardware (its Prior.determinism classifies IntegerExact → passes the gate). The transposed-conv math is unit-tested against hand computation; determinism is structural. Tooling: tools/wai_int_synth_export.py quantizes a synthesis to the integer spec. Caveat (honest): the determinism is free and exact; matching a float model's quality needs quantization-aware training of the shifts — that's an upstream training step, not a decode-contract gap.

P0 still open:

• A cross-hardware byte-equality test needs actual different hardware (CI matrix / multi-arch); the in-repo tests prove same-machine determinism + the structural integer-exactness.

---

Codec menu: register the frontier (ingest, not compete)

#	Gap	Add	Conf	Sink
P1	No standardized neural image codec in the menu	wai.image.jpegai — JPEG AI, finalized as ITU-T T.840.1 / ISO/IEC 6048-1:2025 (Part 1), ~30% over conventional SOTA at equal subjective quality, royalty-free licensing baseline (fits Apache-2.0). Reference VM (encoder+decoder) exists; 4K real-time decode demonstrated on mobile; no browser impl yet	✅ jpeg.org/jpegai	edge/server
P2	wai.neural.bmshj2018 is a 2018 model (oldest/weakest)	wai.neural.mlicpp — MLIC++, −13.39% BD-rate vs VVC/VTM-17 on Kodak (PSNR), linear complexity/memory in resolution	✅ 3-0 ACM	edge/server, ONNX
P2	Keep real-time learned video	wai.neural.dcvc_rt — 125/112 fps @1080p, −21% vs H.266; integerized → can be a real replicate member. Decode orchestration now in Rust (neural/video_dcvc.rs, CPU via ort); GPU is the real-time speed tier	✅ 3-0 arXiv	CPU decode; GPU for real-time

Positioning: the standardization wave (JPEG AI shipped 2025; MPEG/ITU active) is at the codec layer. WAI is the transport/dispatch layer above it — register JPEG AI as a capability, don't compete. WAI's differentiator is the envelope, the capability dispatch, the named-not-shipped prior, and the two honest conformance modes.

---

Perceptual path: a better ruler + stronger generators

#	Change	Detail	Conf	Sink
P3 ✅ DONE	Don't hard-code CLIP cosine	Shipped in wai-rs: ModelRequirement.sim_metric names the embedding space; the trait is SemanticEmbedder (not CLIP-specific), the metric is cosine_sim, and SemanticResult.metric records which space scored. DreamSim (CLIP+OpenCLIP+DINO, human-aligned, MIT) is the recommended ruler — but it's PyTorch-only (no ONNX), which is exactly why the metric is named + sink-supplied, never bundled	✅ DreamSim	source-side light
P4	Register stronger generators as wai.semantic.*	wai.semantic.glc (GLC, latent-space transform, <0.04 bpp natural / <0.01 faces). ⚠️ wai.semantic.perco2 (PerCoV2, SD3, ultra-low bpp) is strong but its paper is CC-BY-NC-SA — non-commercial; register only as an optional, license-flagged profile, never a default for WAI's Apache posture	✅ license / ◐ numbers — PerCoV2 / GLC CVPR'24	server (diffusion)

The CLIP-embed source side is light (browser/edge); the diffusion regen is GPU/server. Tag accordingly.

---

Audio: honest fidelity tiers

• ✅ WavTokenizer (40–75 tokens/sec @24 kHz) is real — but it's extreme-low-bitrate and lossy (weakest reconstruction at ~0.48 kbps). Tag wai.neural.wavtokenizer perceptual, not bit-exact. — arXiv 2408.16532
• ◐ DAC / EnCodec are the higher-fidelity neural-condition codecs at ~6 kbps; only claim replicate for them if integerized (see P0).

---

Browser/edge today vs server-class

• Browser/edge now: floor codecs; bmshj2018 / MLIC++ (small ONNX → WASM/WebGPU); audio token decoders; the CLIP-embed source side.
• Server-class: PerCoV2 / GLC (diffusion); DCVC-RT (real-time, real GPU); SD-XL-Turbo regen.

---

Priority order

1. ✅ P0 integer-exact replicate bar + conformance gate — done in code, and the real member now exists: a trained, activation-calibrated integer model (tools/wai_int_synth_train.py) decodes bit-identically (max|Δ|=0) in Rust. Only the cross-hardware (CI-matrix) byte-equality proof remains.

2. ✅ Real new decoders IMPLEMENTED — the learned-image stack is now three architecture tiers deep:

   • wai.neural.mbt2018_mean — mean-scale hyperprior (neural/hyperprior.rs): per-element (scale, mean) from h_s(z) + Gaussian-conditional rANS. A tier past bmshj2018's factorized prior. Scale-bucketing unit-tested; tools wai_mbt2018_mean_{export,encode}.py.

   • wai.neural.elic — checkerboard context model (neural/context.rs): the JPEG-AI / MLIC++ entropy class. 2-pass decode — anchors (y+x) even with hyperprior params → context net → non-anchors with refined params → g_s. Most of the modern coding gain lives here. Checkerboard partition unit-tested. WAI-native, NOT the ISO JPEG AI bitstream.

   • wai.neural.int_synth — integer fixed-point synthesis (neural/int_synth.rs): bit-exact cross-hardware by construction (see P0). The real replicate member.

   • wai.neural.mlicpp — MLIC++ multi-reference (neural/context.rs sibling mlicpp.rs): channel-autoregressive slices + checkerboard + (sink) global attention — the full −13.39%-vs-VVC entropy class. Decode orchestration implemented (slice loop re-runs the entropy net as context grows); attention nets sink-supplied. Slice×pass coverage unit-tested.

   On the ISO JPEG AI bitstream itself: genuinely cannot be authored without the (paywalled) standard text + reference weights — wai.image.jpegai stays registered for canonical JPEG AI bytes decoded by an external library when one exists. wai.neural.{elic,mlicpp} ARE the JPEG-AI-class architectures, implemented.

Tooling — all decoders now have export/encode: wai_mbt2018_mean_{export,encode}.py (hyperprior), wai_elic_{export,encode}.py (checkerboard — the encode flattens symbols in the decoder's exact anchor→non-anchor order), wai_int_synth_qat.py (quantization-aware training for integer-synthesis quality — the determinism was already proven; this tunes weights+shifts to match a float teacher). All syntax-checked; they need torch + a model to run. 3. ✅ P1 / P2 — superseded by implementation (see STATUS at top): wai.neural.mlicpp has a full Rust decoder (neural/mlicpp.rs, slice×checkerboard orchestration, validated), and mlicv2/cmic route through it. The only registration-without-decoder left is wai.image.jpegai — the ISO bitstream is access-bound (paywalled text + licensee-only reference VM), while its architecture class already runs as elic/mlicpp. 4. ✅ P3 metric-tagged min_sim + metric-agnostic embedder — done in code. 5. ◐ P4 semantic capabilities REGISTERED — wai.semantic.clip (reference perceptual profile) + wai.semantic.perco2 (license-flagged ⚠ CC-BY-NC-SA) added to the registry as Perceptual-tier; generators stay sink-supplied (server-tier diffusion). GLC stays wai.neural.glc (latent coding, Lossy tier). 6. ✅ P5 fidelity tiers — done in code: FidelityTier {Lossless, Lossy, Perceptual} + capability_tier(); every registered capability is tiered, WavTokenizer is Lossy not bit-exact (the research point), wai.semantic.* is Perceptual.

Hard boundary — needs models / tooling, NOT done in-repo (no fabrication)

These three are real ML/library work that can't honestly be written in the Rust crate without external resources. They are registered/ready but not implemented:

• A real integer-exact replicate member — needs a PTQ-integerized decoder (Python + torch + the model). The Rust gate accepts it the moment its determinism string is integer-only.
• MLIC++ decoder (wai.neural.mlicpp) — registered, but its multi-reference entropy model ≠ bmshj2018's, so it needs a real decoder port + an ONNX export of the weights.
• JPEG AI decoder (wai.image.jpegai) — registered, but no Rust JPEG AI library exists yet (2025 standard); needs the reference VM or a port.

Verification status: the research run was rate-limited during its own verify+synthesize pass, but the load-bearing claims were since re-fetched directly from primary sources and upgraded to ✅ — JPEG AI (ISO/IEC 6048-1:2025 + royalty-free baseline), DreamSim (MIT, PyTorch-only), MLIC++ (−13.39% vs VVC), PTQ→integer cross-platform determinism, and PerCoV2's non-commercial license. Remaining ◐ items (GLC's exact numbers, the broader semantic-comms cluster) are single-source, not re-verified.