Transaction-Science/open-standards
Watch
1
0
Fork
You've already forked open-standards
 0
Code Issues Pull requests Projects Releases 17 Packages Wiki Activity Actions 93
open-standards/wai/SPEC.md

47 KiB
Raw Permalink Blame History

WAI v1.0 — Web AI Media Transport & Execution

Status: Draft Standard. Reference implementation: wai-rs/ (Rust lib + cdylib + staticlib, C FFI). Two paths open every conforming file:

  • the neural condition — the sink regenerates the media from a compact conditioning payload against a shared ambient prior (a model). The model is a requirement of existence, never transported and never hash-pinned, exactly as .mp3 does not ship its decoder.
  • the zeroth condition — a menu of registered SOTA standard codecs. Always satisfiable on any device that ships ffmpeg or equivalent. WAI does not define new codecs; it dispatches to the field's best-in-class lossy+lossless libraries (AVIF, JPEG-XL, PNG, Opus, FLAC, AV1, zstd, XZ).

The capability a file requires is named, not supplied. Sinks that have the named capability use it; sinks that don't fall back to a declared fallback capability.

Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119.

1. Scope and model

WAI is a container + capability dispatch standard. It is media- general (image, audio, video, text), royalty-free, and built around the principle that the standard's value is the envelope and the dispatch rules — not in re-implementing existing SOTA codecs.

A WAI file declares (in its manifest):

  • what media it carries,
  • what capability a sink must have to decode it,
  • a fallback capability the sink can use if it lacks the primary,
  • and the conditioning bytes for that capability.

Conforming sinks MUST implement the container envelope (§2), the manifest schema (§3), the capability-dispatch algorithm (§4), and at least the mandatory floor capabilities (§5). Other capabilities (neural and additional zeroth) are OPTIONAL but RECOMMENDED.

2. Container

+--------+----------------+-------------------+----------------+-----------+
| "WAI1" | u32  man_len   | manifest (JSON)   | u32 payload_len| payload   |
| 4 B    | little-endian  | man_len bytes     | little-endian  | bytes     |
+--------+----------------+-------------------+----------------+-----------+
  • Bytes 0..4 MUST be ASCII WAI1. A reader MUST reject any file whose first 4 bytes differ.
  • man_len and payload_len are unsigned 32-bit little-endian.
  • The manifest is UTF-8 JSON, serialized with no insignificant whitespace, preserving key insertion order. Readers MUST NOT depend on key order for semantics; encoders SHOULD preserve order so files round-trip byte-identically.
  • The payload is opaque codec bytes whose interpretation is fully determined by the manifest's model_requirement.capability.

All WAI multi-byte integers are little-endian unless stated otherwise.

3. Manifest schema

{
  "wai": "1.0",
  "media": "image" | "audio" | "video" | "text",
  "intent": "replicate" | "create" | "improve" | "...",
  "model_requirement": { "capability": "<capability-string>",
                         "fallback":   "<capability-string>" | null },
  "conditioning": { "kind": "<codec-id>" },
  "security": { "encryption": "<scheme>" | "none", "kid": <u64> },
  "target":   { ... }
}
  • wai — spec version. This document defines "1.0". Unknown major → reject. Unknown minor → accept by ignoring unknown fields.
  • media — media class; informational. The capability is authoritative.
  • intent — generative intent. replicate (reconstruct the source) is the only intent with normative reconstruction semantics in v1.0. For a neural capability, replicate is sound only under the integer-entropy contract (§7, Replicate-soundness) — a float neural decode MUST NOT claim it.
  • model_requirement.capability — the capability string a sink MUST resolve. See §5 for the registered set.
  • model_requirement.fallback — a second capability the sink MAY use if it lacks the primary. null means the file is inert without the primary.
  • conditioning.kind — short codec id matching the capability. Helps multiplexers route bytes without parsing the full capability string.
  • security — OPTIONAL. Absent, null, or encryption: "none" ⇒ the payload is plaintext (the default). Otherwise the payload is encrypted and MUST be recovered before dispatch; see §6. The capability, media, and conditioning fields always describe the plaintext codec.
  • target — codec-specific reconstruction parameters (dimensions, sample rate, duration, frame count). Informational; the payload is self-describing per the codec's own format.

4. Capability dispatch

A conforming sink MUST implement this algorithm:

  1. Read the manifest. If wai major version is unknown → reject.
  2. Let cap = model_requirement.capability.
  3. If the sink advertises cap, decode payload via the codec registered for cap (§5) and return.
  4. Else, let fb = model_requirement.fallback.
  5. If fb != null and the sink advertises fb, decode via fb's codec and return.
  6. Else the file is inert at this sink — return a clear "missing capability" error to the calling application.

A WAI payload always corresponds to exactly one capability. v1.0 does not support multi-codec multiplexed payloads.

If the manifest declares security.encryption other than "none" (§6), the sink MUST recover the plaintext payload before step 3 and run this algorithm on the plaintext; see §6.4.

5. Registered capabilities (the zeroth menu)

WAI v1.0 registers the following capabilities. Each names a SOTA royalty-free standard codec. The "payload format" column is the on-the-wire payload bytes; in every case it is the canonical bytes that codec normally emits (a JXL file, an Opus stream, etc.) — WAI does not wrap or re-frame them.

Mandatory floor

Every conforming sink MUST implement:

capability payload role
wai.image.png PNG (RFC 2083) universal lossless image
wai.audio.flac FLAC (xiph.org/flac) universal lossless audio
wai.text.zstd zstd (RFC 8478) universal compressed bytes

These three guarantee that any sink with the standard system codec libraries can open some file in every WAI-supported media class.

Recommended modern set

Every conforming sink SHOULD implement:

capability payload role
wai.image.jxl JPEG-XL (ISO/IEC 18181) lossless + lossy, modern SOTA
wai.image.avif AVIF (ISO/IEC 23000-22) lossy SOTA (AV1-based)
wai.image.jpeg JPEG (ITU-T T.81) legacy compatibility
wai.audio.opus Opus (RFC 6716) lossy SOTA
wai.video.av1 AV1 (AOMedia) lossy SOTA
wai.video.av1.lossless AV1 (AOMedia), quantizer=0 mathematically lossless YUV (lossy in RGB due to color conversion)
wai.text.xz XZ/LZMA2 maximum classical text ratio

Neural capabilities (OPTIONAL, sink advertises if its ML runtime supports them)

Each names a model the sink MAY have installed. Payload is a compact conditioning representation specific to the capability (latent tokens, prompt embeddings, etc.). The standard does NOT specify the byte layout of these payloads — the capability owner does.

Neural capabilities are picked on the best-quality-per-bit at each bitrate band principle — not on whatever model is famous or small. 2026 SOTA per medium, with the wire-byte win each one delivers. The decode route for each is implemented in the reference sink (see the source column / neural/*.rs); the model weights are sink-supplied.

Audio:

capability model win vs zeroth source
wai.neural.encodec32 EnCodec at 32 kHz, 4 codebooks ~3 kbps acceptable music facebook/encodec_32khz
wai.neural.dac DAC at 44.1 kHz 6 kbps near-transparent music descript/dac_44khz
wai.neural.mimi Mimi, 12.5 Hz frame rate ~1.1 kbps real-time speech kyutai/mimi
wai.neural.wavtokenizer WavTokenizer single-codebook VQ 0.9 kbps beats DAC @ 9 kbps on UTMOS novateur/WavTokenizer-large-*
wai.neural.snac SNAC multi-resolution RVQ (hierarchical code levels, own sample rate) low-bitrate TTS standard neural/snac.rs

Image:

capability model win vs zeroth source
wai.neural.bmshj2018 bmshj2018-factorized + rANS (byte-exact CompressAI port) ~80× vs raw RGB at q=3 (~0.26 bpp, 29 dB PSNR) CompressAI
wai.neural.mbt2018_mean mean-scale hyperprior (Gaussian-conditional entropy) conditional-entropy family, stronger than factorized neural/hyperprior.rs
wai.neural.int_synth integer fixed-point synthesis (all-i64 transposed-conv, no float/ONNX) bit-identical on every machine — soundly backs intent = replicate neural/int_synth.rs (QAT weights sink-supplied)
wai.neural.int_hyper integer fixed-point conditional codec — whole bitstream→image path integer (integer hyper-decoder σ → integer-σ entropy → integer synthesis) the integer-entropy contract fully assembled; high quality (PTQ of mbt2018_mean) bit-identical on every machine int_hyper_synth.rs (tables sink-supplied)
wai.image.jpegai ISO/IEC 6048 JPEG-AI codestream → RGB — integer entropy front end (mANS) + integer dequant/value path + fixed-order synthesis and post-filters; every per-image quantity is derived from the bitstream, only checkpoint constants are sink-supplied decode-equivalence: one pinned reconstruction of a published-standard codestream to RGB, reproducible on every machine (a portable decoded-pixel hash); entropy and dequant are integer-exact, synthesis reconstructs to within 1 LSB of the reference jpegai_decode.rs (+ jpegai_{bitstream,mans,pipeline,synthesis,icci}.rs)
wai.generate.spectral deterministic integer GENERATIVE synthesis (intent=create) — prompt+seed → integer-LUT spectral composition, no float/ONNX byte-identical on every machine, pinned to (model, prompt, seed, steps); the create-side of "conformance is a hash" (generation-equivalence) generate.rs (small pinned model bundle)
wai.neural.elic checkerboard context model (hyperprior + 2-pass spatial context) the JPEG-AI / MLIC++ entropy class neural/context.rs
wai.neural.mlicpp MLIC++ multi-reference entropy (channel-AR slices + checkerboard + global attention) learned image, low-bitrate high-fidelity research code, neural/mlicpp.rs
wai.neural.mlicv2 MLICv2 / MLICv2+ (2025), MetaFormer transform learned-image fidelity frontier; shares the mlicpp route research code, neural/mlicpp.rs
wai.neural.cmic Content-Adaptive Mamba (ICLR 2026), SSM transform + SCCTX entropy learned-image frontier; shares the mlicpp route arXiv 2508.02192, neural/mlicpp.rs
wai.neural.glc Generative Latent Coding — categorical / VQ entropy <0.05 bpp where JPEG-XL collapses research code, neural/glc.rs

Video:

capability model win vs zeroth source
wai.neural.video_bmshj2018 per-frame bmshj2018 1000×+ vs raw RGB, browser-decodable on WebGPU CompressAI
wai.neural.dcvc_rt DCVC-RT (2025), inter-frame with reference-feature propagation real-time neural video at 1080p/4K research code, neural/video_dcvc.rs
wai.neural.dcvc_fm DCVC-FM (feature modulation), wide single-model quality range (RGB + YUV) SOTA-RD neural video; shares the inter-frame route research code, neural/video_dcvc.rs
wai.video.int_motion DETERMINISTIC INTEGER inter-frame — I-frame + integer block motion-compensated P-frames + integer-quantized residual over rANS, no float/ONNX; motion is whole-pel or half-pel (mv_precision), the latter predicting at the half-grid by integer bilinear interpolation ((a+b+1)>>1, (a+b+c+d+2)>>2) so real footage — which pans a non-integer number of pixels per frame — costs fewer residual bits while staying byte-exact byte-identical frame-by-frame on every machine (frame-state-equivalence, a per-frame portable hash); the temporal extension of the integer contract, where a float inter-frame decode desyncs down the GOP int_video.rs (whole-pel tools/wai_int_video_encode.py; half-pel from real footage tools/wai_int_video_real_encode.py)
wai.video.int_hyper NEURAL-KEYFRAME inter-frame — the I-frame is the LEARNED integer codec int_hyper decoded byte-exact, the P-frames are integer block-MC residuals on top a learned keyframe you can put in a receipt + integer inter-frames; whole decode integer, byte-identical everywhere int_video::decode_pframes ∘ int_hyper_synth::decode_conditional
wai.video.manifest ATTESTED PER-SESSION MANIFEST — the OTT delivery layer, not a codec: a personalised HLS manifest (ads stitched at EXT-X-DISCONTINUITY, blackouts + geo enforced) rendered as a pure function of the session class (content, entitlement, geo, blackout set, ad pod, window) rather than the viewer, and sealed in an Ed25519 receipt that binds the JCP grant (capability, joule ceiling, funds_ceiling = ad budget), the composition, the stitched ads and the policy applied manifest-equivalence: manifest_hash == hash(render(segments)) and session_class_hash are recomputable by anyone holding the bytes, so the ads, blackouts and compose-work stop being dashboard claims; class-determinism means one generation serves every viewer of the class, and uj_per_viewer is the amortised energy that a 1:1-unique generator cannot report below its per-viewer cost. Only joules_micro is measured-attested wai_video_manifest.rs
wai.video.ssai ATTESTED SERVER-SIDE AD INSERTION — a deterministic ad decision (greedy by price within the avail duration and the funds remaining) plus a per-impression Ed25519 receipt binding the candidate pod, the manifest it belongs to, the served creative_hashes and the VAST quartile each reached spend-equivalence: budget_spent_micros is recomputed from the billable impressions and rejected if it exceeds the grant's funds_ceiling — overspend cannot be signed; every creative and quartile is checkable, so a buyer and seller reconcile impressions from the same bytes instead of two logs that only a dashboard joins. Only joules_micro is measured-attested wai_video_ssai.rs
wai.video.keys ATTESTED CONTENT-KEY RELEASE — the DRM licensing decision as a record: every key release, and every refusal, is an Ed25519 receipt binding the KID, the DRM system, the entitlement demanded and the window enforced, with a BLAKE3 key_commit proving which key was released without carrying it decision-equivalence: verify() re-runs the release policy over the receipt's own (grant capability, entitlement, issued, expiry) and rejects any misreported outcome, and enforces granted-iff-committed — so "keys went only to entitled, unexpired sessions" is checkable against the signed log, not asserted. Only joules_micro is measured-attested wai_video_keys.rs
wai.video.package DETERMINISTIC PACKAGING + ATTESTED SEGMENTMAP — a source sliced into content-hashed segments on GOP boundaries; the ordered (index → content_hash) map is a signed Merkle commitment, and identical content (ads, bumpers, replays, simulcast overlap) dedups to one stored/encoded copy map-equivalence + verifiable dedup: segment_map_root lets a wai.video.manifest reference be checked against the package that produced it, and verify() recomputes unique_segments/stored_bytes from the segments so the storage/encode saving cannot be overstated; uj_per_unique_segment is the encode-once energy. Only joules_micro is measured-attested wai_video_package.rs
wai.video.channel VIRTUAL LINEAR / FAST ASSEMBLY — a continuous schedule of assets over wall-clock time, sliced into attested playback windows; windows chain via parent_receipt_hash, so a channel's history is a hash chain of program-as-run records schedule-continuity + recomputable program-as-run: verify() rejects a schedule with a gap or overlap (no dead air, no double-booking) and recomputes aired from the schedule and the window, so what actually aired is checkable by a licensor, advertiser or regulator against the signed chain rather than an operator-held play-out log. Only joules_micro is measured-attested wai_video_channel.rs

Sink-architecture note: the inter-frame neural video codecs (DCVC-RT, DCVC-FM) decode through ort and are CPU-capable for native sinks — their real-time figures come from GPU execution, but the decode is not CUDA-only. They are heavier than per-frame image codecs (a temporal net plus reference-feature state), so a browser sink that cannot host the temporal net should declare only wai.neural.video_bmshj2018 (or another browser-decodable neural video codec) for the video media class. The choice is the deployer's, advertised per sink — the envelope is identical.

TAESD is intentionally absent. TAESD is a distilled SD VAE decoder — it inverts a generative pipeline's encoder, not a general image codec. Outside the SD pipeline it underperforms JPEG-XL at most bitrates while costing ~5 MB of weights and ONNX compute. Don't ship codecs for the brand; ship them for the bit budget.

Image/video scope is honest: zeroth codecs (JPEG-XL, AV1) are already SOTA-adjacent in the bitrate bands where most media lives. Neural image/video capabilities are registered for the bands where zeroth codecs collapse (ultra-low bpp images, bandwidth-constrained streaming) — not as defaults.

Each capability owns its payload format. Decoder implementations (Python source-side, ONNX-export, browser onnxruntime-web) live in tools/wai_<capability>_*.py and wai-web/demo/. New neural capabilities are added by registration; the WAI envelope is unchanged.

See extensions/jwp-receipts.md for the receipt-chain sub-spec covering per-object content hashes, per-group Merkle roots, Ed25519 group signatures, and cross-hop preservation. See extensions/energy-measurement.md for how a receipt's joules_micro is acquired: the required acquisition class (HwShunt / ModelBased / Estimator), the bracketed marginal protocol, and the attribution-quality rule that decides whether a figure may be published at all. See extensions/confidentiality.md for the private-channel sub-spec: MLS (RFC 9420) group keys + SFrame (RFC 9605) object encryption, the byte-level framing behind §6. See extensions/reconstruction-binding.md for the reproducible-reconstruction binding: a C2PA / JPEG-Trust hard binding See extensions/energy-binding.md for the companion energy assertion: the joules a reconstruction cost, bound to that reconstruction, with the acquisition class and coverage C2PA 2.4's c2pa.environmental-sustainability has no field for. over the deterministically decoded pixels of a neural-coded asset (SHA-256 over the canonical RGB plus the descriptor that pins the reconstruction) — the reconstruct-side companion to provenance, applicable to wai.image.jpegai and the byte-exact wai.neural.int_* codecs.

Interactive, temporal & computed content (extensions)

Media that consumes input, moves over time, or is computed from a compact instruction set. Each registers its own payload format and a deterministic *-equivalence conformance criterion over the wai.det.fixed64 fixed-point floor — the reconstruct half of the standard. See the linked extension for each.

capability payload conformance · extension
wai.world.scene / .replay / .live WWLD container (op log) replay-equivalence · interactive-worlds
wai.world.model generative-world transport: a signed action trace (the world CausalOp log) + a declared model_id (world_model.rs) the wire carries the action trace, not the pixels — it replays under canonical total order to one session hash on every machine (replay-equivalence), the generated frames vouched for by a signed, joules-accounted provenance receipt. The neural sibling of .replay; with an int_* renderer the frames hash-converge too (a generative interactive world you can put in a receipt)
wai.mesh.glb glTF 2.0 Binary (Khronos) asset, content-addressed (decoder sink-supplied)
wai.splat.spz SPZ (Niantic) asset, content-addressed; presentation only
wai.audio.scene WSCN scene (objects + trajectories) mixdown-equivalence · spatial-audio
wai.audio.score WSCR container (note events) music as instructions; sink synthesizes, mixdown-equivalence · score
wai.audio.int_codec LEARNED 1-D transform codec — integer transposed-conv synthesis + integer IGDN over a factorized-rANS latent (model + bitstream from tools/wai_int_audio_export.py) sample-equivalence: int16 PCM byte-identical on every machine (a portable PCM hash); the audio analog of wai.neural.int_hyper, where a float learned-audio decode desyncs on ~1e-5 round-off. Pure i64, no float/ONNX (int_audio.rs)
wai.haptic.signal WHAP envelope (intensity/sharpness keyframes) sample-equivalence · haptics
wai.feed.state WWLD, sim wai.world.sim.feed0 replay-equivalence · state-feeds
wai.avatar.pose WWLD, sim wai.world.sim.skeletal0 pose-equivalence · avatars
wai.splat.4d W4DS container (per-Gaussian keyframed tracks) frame-state-equivalence · volumetric
wai.splat.int_codec LEARNED Gaussian-splat ATTRIBUTE codec — a learned transform over the [14,S,S] attribute image (pos/scale/rot/color/opacity), integer synthesis (int_hyper machinery, 14 channels) over a factorized-rANS latent (int_splat.rs; model + bitstream from tools/wai_int_splat_export.py) splat-equivalence: the decoded attribute tensor is byte-identical on every machine (a portable cloud hash) — the int_hyper of 3D, where a float splat-attribute decode desyncs. The float rasterizer stays sink-supplied presentation; the attributes are the contract. Pure i64, no float/ONNX
wai.film.linear WFLM container (sealed world + camera track + frame schedule) film-equivalence · linear-film
wai.film.reel WREL container (sealed world + a sequence of shots = camera track + schedule each) reel-equivalence (film-equivalence with cuts) · linear-film
wai.ternary.tt WQTT container (balanced-ternary MPS cores — packed trits + integer per-column scales) a quantum-inspired (tensor-network) transform ported from the ternary/MPS fabric with ONE change: the per-column scale is an integer, so the whole decode is i64/i128 and the reconstruction is byte-identical on every machine — reconstruction-equivalence (a portable recon hash), receiptable. Honest scope: a provenance transform, NOT a rate-competitive codec (measured RD is mediocre — Lossy tier; see the extension's §5 verdict). Encoder-side float SVD only · ternary-tt
wai.quantum.circuit WQC container (gate op-log — Clifford+T + dyadic phase) the wire carries a circuit (kilobytes, linear in gate count), the sink CLASSICALLY simulates it in a fixed-point complex floor (Amp = i64 re/im), and the reconstructed 2^n-amplitude statevector is byte-identical on every machine — statevector-equivalence (a portable statevector hash); measurement is sampled by the pinned splitmix64 so a shot histogram at a pinned (seed, shots) converges too (shot-histogram-equivalence). The dyadic phase ladder gives the QFT ("quantum FFT"), transported as a circuit and reconstructed byte-exact. No quantum hardware — NISQ noise breaks the determinism the standard rests on; this makes a quantum computation transportable + receiptable, with no speedup claimed · quantum-sim

And the verify half — a signed, energy-accounted lineage for opaque captured content WAI does not reconstruct (content-addressing + receipts + measured joules): extensions/provenance.md (wai.asset.provenance).

The same verify half, specialized to quantum operations — a signed, content-addressed, joule-metered, grant-authorized receipt of a calibration run (wai.quantum.calibration) or a dispatched QPU job (wai.quantum.job), where a job binds the exact calibration state it ran under and verify() enforces the grant's joule ceiling (gate ⊗ meter ⊗ receipt in one object). WAI owns the receipt; the sink's control system runs the hardware: extensions/quantum-ops.md.

Immersive audio, cinema, volumetric, haptics, HDR (registered, sink-supplied)

The same "name the capability, the sink supplies the decoder" pattern extends to the standard carriers of the immersive / theatrical / volumetric media a full distributor needs. Each names an open reference where one exists; the proprietary and SEP-pool entries are flagged, never defaults. Payload is the named standard's canonical bytes.

class capabilities
immersive audio wai.audio.iamf (IAMF/Eclipsa, royalty-free) · wai.audio.ac4 (Dolby AC-4, proprietary) · wai.audio.ivas (3GPP IVAS, SEP-pool)
cinema mastering wai.image.jp2 · wai.video.j2k (JPEG 2000 / DCP, OpenJPEG)
volumetric / 3D wai.mesh.draco (royalty-free) · wai.pc.gpcc · wai.pc.vpcc (MPEG point clouds) · wai.splat.sog (Gaussian-splat compression, candidate)
haptics wai.haptic.mihs (MPEG-I Haptics, ISO/IEC 23090-31; its 23090-33 ships bit-exact conformance vectors)
HDR dynamic metadata wai.meta.hdr10plus (royalty-free) · wai.meta.dovi (Dolby Vision, proprietary) — companion descriptors to a video payload, not standalone media

Every registered capability also carries an IP class — codecs::capability_license(cap) ∈ {royalty-free, patent-pool, proprietary, non-commercial}, with is_royalty_free(cap) as the predicate. It is a registry property (not a per-file manifest claim — a codec's licence can't be honestly asserted in an envelope's header), so a deployer can audit a title or a whole catalogue in code: encode only to royalty-free capabilities, or flag every pool/proprietary one. The per-title choice of which toll road to pay is a property of the format, enforceable, not a policy on a slide.

See research/media-codec-registry.md for the per-capability standard / spec number / verified royalty status / reference library and the remaining spec follow-up (companion-component binding for the HDR / audio metadata descriptors).

6. Confidentiality (OPTIONAL — encrypted payloads / private channels)

By default a WAI payload is plaintext: any sink that advertises the named capability (§4–§5) can decode it. This is correct for public media — a shared world, a broadcast film. Some channels are instead private: only an intended group may recover the content, and any relay or multiplexer that carries it MUST be able to route, meter, and attest the bytes without reading them.

Confidentiality is an OPTIONAL layer that is orthogonal to the capability. It wraps the payload bytes; the capability still names how to decode the plaintext once recovered. A sink decrypts first, then runs capability dispatch (§4) on the plaintext.

6.1 Manifest field

A manifest MAY carry a security object:

"security": { "encryption": "mls-sframe", "kid": 0 }
  • security.encryption — encryption scheme id. Absent, null, or "none" ⇒ the payload is plaintext (the default; §1–§5 unchanged). "mls-sframe" ⇒ the payload is an SFrame (RFC 9605) ciphertext object whose group key is agreed by MLS (RFC 9420) — the draft-jennings-moq-e2ee-mls binding.
  • security.kid — the SFrame key id selecting which sender/epoch key opens the object. It is also carried in the SFrame header; the manifest copy lets a multiplexer route without parsing the payload.

media, intent, conditioning.kind, and model_requirement.capability continue to describe the plaintext codec. They are NOT encrypted and remain visible for routing — confidentiality protects the content, not the shape.

6.2 Keys are a requirement of existence

Exactly as the model behind a neural capability is never transported (§1), the group key is never transported in the WAI file. Group membership is established out of band via MLS; each member derives the 32-byte SFrame base key from the group's exporter secret under the label "moq-jwp sframe base key". A sink that is not a group member cannot derive the key and the file is inert at that sink — the confidentiality analog of a missing capability.

6.3 Relay transparency and attestation

A WAI relay or multiplexer that carries an encrypted payload MUST treat it as opaque: it MUST NOT require the plaintext to route, meter, or attest, and MUST NOT attempt capability dispatch on ciphertext. The JWP receipt chain (extensions/jwp-receipts.md) commits over the ciphertext content hash, so per-object delivery, energy, and integrity are attested end-to-end without any relay holding the key. Content secrecy (SFrame/MLS) and delivery attestation (JWP) are independent layers over the same bytes and compose: a relay proves it forwarded the exact sealed bytes it was given while never learning what they say.

6.4 Decrypt-then-dispatch

A confidentiality-aware sink MUST, before running §4:

  1. If security.encryption is absent / null / "none", proceed to §4 on the payload unchanged.
  2. Else, if the sink does not implement the named scheme, or holds no key for security.kid, the file is inert at this sink — return a clear "no key / unsupported encryption" error. It MUST NOT feed ciphertext to a codec.
  3. Else recover the plaintext payload via the named scheme (for mls-sframe: SFrame open under the member's base key), then run §4 on the recovered plaintext.

A sink that does not implement confidentiality at all MUST refuse an envelope whose manifest carries a non-none security.encryption cleanly, rather than mis-decode ciphertext.

Byte-level SFrame framing, the SFrame key schedule, and the MLS binding (including per-sender keying for multi-sender channels) are specified in extensions/confidentiality.md. The reference implementation is the moq-sframe (SFrame) and joule-comm-mls (MLS exporter → base key) crates, with browser bindings in joule-comm-mls-wasm; Track::e2e_encrypted is the switch that selects this path. Live: https://wai.transaction.science/private-channel

7. Conformance

  • A conforming sink MUST: implement the container (§2), capability dispatch (§4), and the mandatory floor (§5).
  • A conforming encoder MUST emit containers (§2) that any conforming sink can read; the payload MUST be the canonical byte stream for the registered capability.
  • Conformance is verified by bit-exact decode-equivalence with the reference library named in §5 for each registered capability. The WAI Rust reference impl in wai-rs/ is built on those very libraries and ships a test suite that exercises every registered capability through the full WAI envelope (cargo test --lib).

Replicate-soundness — the integer-entropy contract (normative)

intent = replicate claims the sink reproduces the exact source bytes. For a neural capability that claim is only sound if the decode is byte-identical on every machine — and that is not automatic. A learned codec's entropy decode desyncs the instant the decoder's scale-bucket / σ computation drifts from the encoder's by one float ULP: the bucket assignment flips and the rest of the stream corrupts. So a neural decode that runs float inference is reproducible only on the same build, never guaranteed across hardware.

Therefore a neural capability MAY back intent = replicate only if its entire decode path — entropy parameterization and synthesis — is integer / fixed-point, with no float operation anywhere (the integer-entropy contract). The reference impl factors the halves into ORT-free, pure-i64 modules a codec composes: int_entropy (the deterministic scale→bucket step that removes the float-σ desync), int_synth (integer synthesis), and the integer rANS — assembled into a full conditional decode by int_hyper_synth (wai.neural.int_hyper: factorized rANS-decode z → integer hyper-decoder h_s → σ,μ → integer-σ bucket → rANS-decode residual → y_hat = r + μ → integer synthesis — both bitstreams entropy-coded, no float anywhere). This is a property of the capability, classified in the registry as codecs::capability_replicate_class(cap) -> ReplicateClass:

  • NeuralIntegerExact — integer by construction; byte-identical on every machine; the only neural class that soundly backs cross-hardware replicate. (wai.neural.int_synth is synthesis only; wai.neural.int_hyper is the full conditional codec — σ source, entropy, and synthesis all integer.)
  • GenerativeDeterministic — the create-side of byte-exact conformance. A generator (intent = create) whose entire pipeline is integer/fixed-point and pinned to (model, prompt, seed, steps), so the generated content is byte-identical on every machine. There is no original to reconstruct, yet the output hash is portable and receiptable — generation-equivalence. The fixed-point sampler is mandatory: a float RNG or float sampling step desyncs the stream exactly as a float σ does. (wai.generate.spectral: a pure-integer spectral synthesis — prompt+seed draw integer-LUT sinusoid components by a splitmix64 PRNG; the float sin() that bakes the LUT runs once at model-build, never at generation.) Distinct from Behavioral, which conforms by similarity and never reproduces bytes.
  • NeuralFloat — runs float inference; exact only on the same build. An encoder MUST NOT emit intent = replicate for one of these; perceptual / "exact on this build" is the honest claim.
  • StandardDefined — reconstruction is fixed by the named codec/standard (the floor + menu codecs) or by WAI's deterministic floor (§ extensions); the neural float hazard does not apply.
  • Behavioral — a sink-generator path that conforms by similarity, not bytes (wai.semantic.*).

A conforming sink that receives intent = replicate for a NeuralFloat capability MUST refuse the replicate claim (decode as perceptual or reject). The reference impl enforces this per-envelope in decode_envelope_strict, which rejects a replicate envelope whose pinned prior is not integer-exact (DeterminismClass) — and a NeuralFloat capability cannot honestly carry an integer-exact prior. The registry class is the static audit of the same property: wai capabilities prints it per row and reports the count of neural integer-exact capabilities, so a deployer can require "dispatch replicate only to a cross-hardware-sound decode" in code, exactly as it audits royalty status.

This is the standard fixing the one axis the learned-codec field leaves implicit: a neural codec that is integer all the way down is the only one that can put a reconstruction in a receipt and have it verify on someone else's machine.

Reproducibility is a spectrum, not a single point, and the ReplicateClass values above span it. See extensions/determinism-tiers.md, which names the two tiers a neural capability can satisfy: entropy-consistency — the entropy decode reproduces across devices (integer entropy arithmetic or explicit parameter transmission), the guarantee the base learned-codec standards make and the floor for a sound neural replicate; and decode-equivalence — the whole decode to the final medium reproduces, so the rendered result has a portable hash, the tier WAI adds on top and the precondition for a hard reconstruction binding.

Machine-checked totality (informative)

The cross-architecture matrix establishes byte-exactness by sampling: it decodes fixed vectors on five real OS/arch combinations. wai-verify/ supplies the complementary half by reasoning — a model checker (Kani/CBMC) over the same #[path]-included sources, for every input in a bounded window.

What it discharges is the property the byte-exactness claim rests on: arithmetic totality. Rust integer overflow panics in debug and wraps in release, so an overflowing decoder is not one implementation but two, and which one a sink runs depends on how it was compiled. A decode proved free of overflow, out-of-bounds access and non-termination is a decode whose bitstream denotes exactly one symbol sequence.

matmul's accumulation is total for fixed-point operands of realistic magnitude (±2^31 at Q16). This one is a conditional result and is stated as such: the product is formed in i128 then truncated to i64 and accumulated with a plain +=, which genuinely does overflow for large enough operands. Nothing in the signature constrains magnitude, so callers inherited that safe-operating range silently; the harness turns it into a written, checked precondition rather than reporting an unstated contract as a defect.

Currently proved for the integer transformer kernels (int_transform): softmax is total over the whole fixed-point path (ln2 reduction, i128 polynomial, shift, division) and its output sums to exactly 2^PROB_FRAC with no negative probability — the normalisation the entropy stage depends on, since a denormalised distribution becomes a denormalised CDF. The checker also shows the degenerate uniform fallback to be unreachable; that branch would violate the same invariant if it ever fired, so the property currently holds by unreachability rather than by construction, and the code says so at the site.

Currently proved for the scale→bucket primitive (int_entropy): det_build_index always returns an index addressing a real scale-table entry, and is monotone in σ — a larger σ never selects a smaller bucket. The monotonicity is proved for an arbitrary edge table rather than only an ascending one, so it does not rest on a precondition a caller might not maintain; it is the property the bucketing exists to guarantee, and a non-monotone bucketing would mis-order the table without ever panicking.

Currently proved for the integer rANS coder: RansDecoder::new is total for every input slice; decode_categorical is total and returns an in-alphabet symbol for all admissible CDFs and all streams in the window; and decode's bypass escape is total. Each harness carries a kani::cover obligation, so a proof cannot pass vacuously on assumptions that admit no state.

This is not redundant with the matrix. The matrix decodes well-formed vectors and therefore cannot see defects only a malformed stream reaches; four such defects in the bypass escape — a stream-controlled shift distance, two saturating-arithmetic gaps, an unbounded read — were found this way and fixed, with the conformance vectors confirming the fix changed nothing a conforming encoder can produce.

The wai.video.ssai decision policy is test-backed, not proof-backed: the natural theorem — that decide never proposes a fill exceeding the funds, the avail duration or the slot count — did not close (three configurations, up to 900s each). The obstacle is that the decision sorts on a heap-allocated key, so the model carries memcmp and the allocator machinery rather than the budget arithmetic. Stubbing that away would prove a property of the stub. What is proved there is narrower: a fresh decision bills nothing until its beacons arrive, so a fill cannot be mistaken for delivery.

layernorm closes only at ±2^10 — below realistic Q16 activation magnitude (at ±2^20 it did not close in 25 minutes). Its guarded divide and surrounding arithmetic are therefore established for small operands only, not over the kernel's working range. The obstacle is isqrt over a symbolic u128, which is a limit of bounded model checking rather than a property of the code. attention, linear, igdn and the composed TransformerBlock are not covered at all. The technique does not reach everything, and this section records where it stops rather than listing only what it reached.

Scope, stated plainly: these are bounded proofs — small symbolic CDFs, fixed-length symbolic streams, explicit unwind limits — not universal ones, and they cover the stream-controlled paths rather than the whole crate. A CDF supplied by internal tables is still assumed well-formed. The verification claims what it checks and no more.

7.1. Multi-rendition envelope (v1.1 extension)

The v1.0 single-payload envelope is sufficient when one source produces one rendition for one capability. v1.1 adds an OPTIONAL multi-payload form for cases where the source emits the SAME content in multiple renditions and lets the sink pick — typically because the sink's compute budget, bandwidth budget, or runtime preference (browser- native vs WebGPU vs WebNN vs native ML) differs at decode time.

The model_requirement.fallback field in v1.0 is declarative — it names a capability the sink CAN look for. v1.1 makes fallback (and multi-rendition selection generally) executable by carrying every declared capability's payload in one envelope.

Self-contained-ecosystem note

In a controlled-deployment ecosystem (every sink ships from one authority and therefore has a known capability set) the multi-rendition form is NOT a "graceful fallback for missing capability" — every sink has every required capability. Renditions exist for policy selection: high-quality on a desktop sink with WebGPU; low-quality on a phone sink; quality-vs-latency by bandwidth headroom; native vs WebNN by which runtime is initialized. The sink picks by deployer-defined policy, not by which capability happens to be installed.

v1.1 wire format

+--------+----------------+-------------------+----------------+----...---+
| "WAI2" | u32  man_len   | manifest (JSON)   | rendition table| payloads |
+--------+----------------+-------------------+----------------+----...---+
  • Magic is WAI2 (distinct from v1.0's WAI1; sinks dispatch by magic).
  • Manifest is JSON with a single renditions array of self-contained rendition entries, in deployer-preferred order: 
    {
  "wai":    "1.1",
  "media":  "audio",
  "intent": "replicate",
  "renditions": [
    { "capability": "wai.neural.encodec32", "kind": "encodec_tokens" },
    { "capability": "wai.audio.opus",       "kind": "opus"           }
  ],
  "target": { "sr": 48000, "dur": 5.0 }
}
    Each entry has its own capability (the dispatch key) and kind (the codec id matching the bytes in the corresponding payload).
  • Rendition table: u16 n_renditions | n × (u32 offset, u32 length), offsets relative to the start of the payloads block. The table's i-th entry corresponds to renditions[i] in the manifest.
  • Payloads block: the rendition byte ranges, packed contiguously.
  • n_renditions MUST equal the manifest's renditions.length.

Selection algorithm

A v1.1-aware sink MUST:

  1. Read the manifest, get the ordered list of model_requirement entries.
  2. Walk the list in order; pick the first entry whose capability the sink either advertises OR whose declared backend the sink prefers under its deployer policy.
  3. Read the corresponding rendition's bytes from the payload block.
  4. Dispatch as in v1.0 §4.

A v1.0 sink that encounters a WAI2 envelope MUST refuse it cleanly (unknown major). No silent fallback to the first payload — that would hide a major-version mismatch.

Companion components (descriptors bound to a payload)

A WAI2 component carries an OPTIONAL role. Absent ⇒ "rendition" (a selectable alternative of the same content — the default above). "companion" marks a descriptor applied with the chosen rendition, not instead of it, naming the component it binds to in companion_of. This resolves the one media class that is not a standalone payload: HDR dynamic metadata (wai.meta.hdr10plus / wai.meta.dovi, SMPTE ST 2094) and immersive-audio metadata accompany a video/audio payload rather than replacing it. Carried as companions, the descriptor and its target ride in the same envelope, so the binding is intrinsic — not a spoofable sidecar.

The selection algorithm is unchanged: a sink picks a rendition among the role != "companion" entries, then applies every companion whose companion_of names the chosen component (e.g. the HDR10+ block for the AV1 video the sink picked). pick/primary never return a companion. Static HDR characteristics (ST 2086 mastering display, HLG transfer) stay in-band in the video bitstream / a manifest color field and need no companion.

v1.0 compatibility

v1.0 envelopes (WAI1) remain valid and are unchanged. Encoders SHOULD emit WAI1 when only one rendition is present (simpler form, wider sink support). WAI2 is for sources that genuinely benefit from shipping multiple renditions per file.

8. Versioning

  • The wai manifest field is MAJOR.MINOR. A reader MUST reject an unknown MAJOR. A reader MUST accept an unknown MINOR by ignoring fields it does not understand, provided MAJOR matches.
  • The container format (§2) and the manifest schema (§3) are frozen for all of 1.x.
  • New capabilities (§5) MAY be added in any MINOR revision. Existing capabilities MUST NOT be redefined — once a capability string is registered for a codec, its payload format is fixed forever.
  • Removing or repurposing an existing capability requires a new MAJOR.

Appendix A — Reference implementation

wai-rs/ (Rust). Builds as lib, cdylib, staticlib so the C ABI in wai-rs/src/ffi.rs drops in as libwai.dylib / .so / .dll and can be called from any language. The crate wraps:

  • image: image crate (PNG, JPEG, AVIF via ravif), jpegxl-rs (libjxl 0.11.x).
  • audio: opus (libopus 1.x), claxon + flac-bound (libFLAC).
  • video: rav1e (AV1 encode) + dav1d (AV1 decode).
  • text: zstd (libzstd), xz2 (liblzma).

Run cargo test --lib (with RUSTFLAGS="-L /opt/homebrew/opt/flac/lib" on macOS, or via the committed .cargo/config.toml).

Appendix B — Why a capability menu, not a custom codec

Earlier drafts of WAI defined custom transform + entropy stages for each medium. That direction was dropped: re-implementing JPEG / MPEG / Opus poorly is strictly worse than calling the field's mature libraries. WAI's value is the envelope + capability dispatch + the neural-shared-prior model, all of which are absent from existing standards. The zeroth condition's purpose is availability, not codec novelty.

Appendix C — License

Apache-2.0. The standard and reference implementation are open; royalty-free implementation is a precondition for adoption (avoiding the H.264/H.265 royalty trap; following the VP9/AV1/AVIF/JPEG-XL open- codec precedent).