open-standards/wai/wai-rs/tests/vectors/neural_audio
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Transaction Science 32b148cd75
Some checks failed
wai-quantum energy-landscape jobs / does this change touch it (push) Successful in 14s
WAI breaking changes and the conformance runner / WAI breaking · does this change touch WAI (push) Successful in 18s
WAI breaking changes and the conformance runner / WAI breaking · wire, record and API against main · linux (push) Successful in 1m43s
wai-audit collector and player integration / does this change touch it (push) Has been cancelled
wai-audit collector and player integration / collector · linux (push) Has been cancelled
wai-audit collector and player integration / collector, player integration and the browser end to end · macos (push) Has been cancelled
WAI operation catalogue / does this change touch it (push) Has been cancelled
WAI operation catalogue / the catalogue and every door · linux (push) Has been cancelled
wai-quantum for JavaScript / does this change touch it (push) Has been cancelled
wai-quantum for JavaScript / javascript · linux (push) Has been cancelled
repository checks / lockfiles, advisories, tests in CI, workflow hygiene, spec and section references, public text (push) Has been cancelled
wai-quantum studio and desktop / does this change touch it (push) Has been cancelled
wai-quantum studio and desktop / studio · browser (push) Has been cancelled
wai-quantum studio and desktop / desktop app · macos (push) Has been cancelled
wai-quantum studio and desktop / desktop app · linux (push) Has been cancelled
wai-quantum studio and desktop / desktop app · windows (push) Has been cancelled
wai-sink-android native sink library / does this change touch it (push) Has been cancelled
wai-sink-android native sink library / the JNI layer and the platform-free half on the host JVM · linux-arm64 (push) Has been cancelled
wai-sink-android native sink library / every ABI, the AAR, and the instrumented tests in a headless emulator · macos (push) Has been cancelled
wai-sink-swift native sink package / does this change touch it (push) Has been cancelled
wai-sink-swift native sink package / libwai slices, desktop tests, simulator tests · macos (push) Has been cancelled
WAI Studio / does this change touch it (push) Has been cancelled
WAI Studio / the catalogue's actions, the unit tests and the core · linux (push) Has been cancelled
WAI Studio / Studio in Chromium and Firefox · macos (push) Has been cancelled
WAI Studio / Studio in WebKit · linux (push) Has been cancelled
wai-quantum energy-landscape jobs / the module under wasm32-wasip2 (wasmtime) (push) Successful in 2m33s
joule-moq-remote host agent / joule-moq-remote · does this change touch it (push) Has been cancelled
joule-moq-remote host agent / joule-moq-remote, every target (linux-arm64) (push) Has been cancelled
joule-moq-remote host agent / joule-moq-remote, every target (macos) (push) Has been cancelled
joule-moq-room room service / joule-moq-room · does this change touch it (push) Has been cancelled
joule-moq-room room service / joule-moq-room, every target (push) Has been cancelled
joule-moq-web browser client / joule-moq-web · does this change touch it (push) Has been cancelled
joule-moq-web browser client / joule-moq-web · browser end-to-end (macos) (push) Has been cancelled
wai-app and Create / wai-app · does this change touch it (push) Has been cancelled
wai-app and Create / wai-app · tests, clippy, the line language's verifier (linux) (push) Has been cancelled
wai-app and Create / wai-app · Create, browser end-to-end (macos) (push) Has been cancelled
wai-fleet service / does this change touch it (push) Has been cancelled
wai-fleet service / service · linux (push) Has been cancelled
wai-fleet service / service · macos (push) Has been cancelled
wai-ledger service / does this change touch it (push) Has been cancelled
wai-ledger service / service · linux (push) Has been cancelled
wai-ledger service / service · macos (push) Has been cancelled
wai-quantum energy-landscape jobs / the same bits · macos (push) Successful in 4m6s
wai-quantum energy-landscape jobs / native against WebAssembly, over many seeds (push) Successful in 4m38s
WAI breaking changes and the conformance runner / WAI breaking and conform · their tests, and the independent verifiers through wai conform · linux (push) Successful in 5m17s
wai-key-proxy / does this change touch it (push) Has been cancelled
wai-key-proxy / proxy · linux (push) Has been cancelled
wai-quantum energy-landscape jobs / the same bits · linux-arm64 (push) Has been cancelled
wai-quantum energy-landscape jobs / reader, kernels, answers and the stated result · linux, release (push) Has been cancelled
wai: EnCodec 24/48 kHz and DAC 16/24 kHz registered with a pinned payload; SoundStream a candidate
The neural audio models WAI did not yet name, each at the operating points of its own
publication or release, royalty-free (EnCodec's repository and DAC's weights are MIT),
neural-float, one ONNX file each:

- wai.neural.encodec24: mono, 2/4/8/16/32 codebooks of 1024 at 75 Hz (1.5 to 24 kbps)
- wai.neural.encodec48: stereo, one-second chunks overlapping by 1% with a scale each,
  2/4/8/16 codebooks at 150 Hz (3 to 24 kbps)
- wai.neural.dac16: 1 to 12 codebooks at 50 Hz; wai.neural.dac24: 1 to 32 at 75 Hz
- wai.neural.soundstream: a candidate. The publication's models were not released, and
  the open speech codec built on its architecture ships its own non-ONNX models and
  packet, so no payload format or parameter set is pinned; never dispatched.

SPEC §5 gains *Payloads of the neural audio codecs*: the payload (the layout the route
already read, and a chunked form for the 48 kHz model), the checks made before the
decoder runs, and the decoder contract: the model's full codebook count always, a row
of 1024 marking a codebook not used, so one file decodes every bitrate. DAC's 16/24 kHz
decoders give t*320 - 8 samples, and the bound says so.

neural/audio.rs checks the payload, fills the unused rows, decodes chunk by chunk and
joins the 48 kHz chunks by the model's linear overlap-add; DecodedAudio gains
`channels`, two interleaved. The C ABI has no two-channel form, so encodec48 is not
advertised through it and is refused before decoding.

tools/wai_neural_audio_export_onnx.py exports each decoder from the released weights
and checks it against the model's own decode with the first q codebooks: on Thor all
four agreed within 1.5e-5 at every operating point tested. Vectors in
wai-rs/tests/vectors/neural_audio/ (stdlib generator, its own Python decode as the
second implementation); tests/neural_audio_vectors.rs decodes each through the dispatch
within 1e-5 of it and refuses a payload outside its model, an unpinned decoder, the
candidate and two channels through the C ABI; the neural gates workflow runs it.
Registry exports, wai.h, the glossary links and pin_verify.py's one-file set follow.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-09 09:40:50 -05:00
..
dac16 wai: EnCodec 24/48 kHz and DAC 16/24 kHz registered with a pinned payload; SoundStream a candidate 2026-11-09 09:40:50 -05:00
dac24 wai: EnCodec 24/48 kHz and DAC 16/24 kHz registered with a pinned payload; SoundStream a candidate 2026-11-09 09:40:50 -05:00
encodec24 wai: EnCodec 24/48 kHz and DAC 16/24 kHz registered with a pinned payload; SoundStream a candidate 2026-11-09 09:40:50 -05:00
encodec48 wai: EnCodec 24/48 kHz and DAC 16/24 kHz registered with a pinned payload; SoundStream a candidate 2026-11-09 09:40:50 -05:00
.gitattributes wai: EnCodec 24/48 kHz and DAC 16/24 kHz registered with a pinned payload; SoundStream a candidate 2026-11-09 09:40:50 -05:00
README.md wai: EnCodec 24/48 kHz and DAC 16/24 kHz registered with a pinned payload; SoundStream a candidate 2026-11-09 09:40:50 -05:00

Neural audio route vectors

One vector for each capability whose payload SPEC §5 pins (Payloads of the neural audio codecs): wai.neural.encodec24, wai.neural.encodec48, wai.neural.dac16 and wai.neural.dac24. They hold the route of the reference sink (neural/audio.rs) to the pinned layout and decode; tests/neural_audio_vectors.rs decodes them.

Provenance. Generated by tools/wai_neural_audio_vectors.py from fixed dyadic parameters and SplitMix64 codes (seeds 1 to 4). No audio and no model weights: each decoder has its model's geometry and contract, not its network. Apache-2.0, as the repository.

Each directory holds:

  • decoder.onnx — a small decoder with the model's codebook count Q, codebooks of 1024, a hop of 320 and its channels: inputs audio_codes (int64 [1, 1, Q, T]) and audio_scales (float [1]), output audio_values (float [1, C, T·320 − d], d = 8 for the two DAC models, else 0). A code c of codebook i adds c·A[i] + B[i] to its frame's latent z, a row holding 1024 adds nothing, and sample k of the frame on channel ch is tanh(z·W[ch][k]) times the scale. A, B and W are dyadic, so every value before the tanh is exact in binary32.
  • vector.wai — a WAI1 envelope of the capability, pinned to decoder.onnx, whose payload carries fewer codebooks than the model holds (all 32 for dac24), so the route must mark the others unused; encodec48 carries two chunks with scales 0.75 and 0.5, its last chunk six frames.
  • expected.f32 — the decode at the samples meta.json selects (every one, or for encodec48 every sixteenth and every one in the chunks' overlap), each sample's channels in order, as little-endian binary32. The generator computes it itself, in Python, from the payload and the decoder's parameters: the second implementation of the route.
  • meta.json — the capability, its geometry, the payload's shape, the selection rule and the decoder's SHA-256.

The reference's decode is within 10⁻⁵ of expected.f32 (the decoder's tanh is the runtime's). python3 -I tools/wai_neural_audio_vectors.py --check rebuilds every file and compares: the decoders and envelopes byte for byte, expected.f32 within 10⁻⁶.