# Neural voice packs `build_voices.py` builds the optional on-device neural voices for the AAC Reference App: one runtime (Moonshine WASM 0.1.5), the shared Kokoro-82M core, the G2P data for each language, and the voices themselves. It writes them in the `aac-voice-index/1` pack format with a catalogue and full licence texts. Neither the pack files nor the app bundle go into git. The catalogue and `LICENSES.md` are committed in `data/voices/`; each voice's sample and the measurements its rating rests on are committed here. v1 offers 16 voices in 8 of the 45 languages: en, es, fr, pt, it, zh and hi on Kokoro, and nl (Flemish) on Piper. ha and yo are `planned` for v1.1 (OpenBible VITS on onnxruntime-web). de, ja and uk had voices, which are withheld because they say other words (below); with the other 32 languages, they are `web-speech-only`, each with its reason in the catalogue and in `LICENSES.md`. ## Ratings, and voices not offered A voice must say the words the person chose. That is measured, not assumed: `intelligibility.py` has every voice say its language's seven everyday phrases (`phrases.json`: a request, help, stop, yes, no, a feeling, a question), a speech recognizer (Whisper large-v3-turbo) writes down what it hears, and the score is how much of each phrase it got back (0 to 1, ignoring case, spaces and punctuation). The results are committed in `intelligibility.json`, and `build_voices.py` derives each voice's `rating` from them and from the upstream naturalness grade (Kokoro's VOICES.md), by `RATING_RULE`: | rating | earned by | the app says | |---|---|---| | withheld | under 0.80 on average: it says other words | (not offered) | | `experimental` | under 0.90 on average, any phrase under 0.60, or never measured | "Experimental: it may not sound right. Listen to it before relying on it." | | `clear` | 0.90 and over | "Clear and easy to understand, though less natural." | | `natural` | 0.95 and over, with an upstream grade of A or B | "Sounds natural." | A missing or unknown rating reads as `experimental` in the app. Each pack carries its `intelligibility` (mean, min, model, date) for maintainers, beside `quality`, the notes behind the grade; neither is shown. Measured on 2026-09-30, twice (Piper voices vary a little from run to run): | voice | mean | lowest | rating | |---|---:|---:|---| | en Heart (A) | 1.00 | 1.00 | natural | | en Michael (C+) | 0.99 | 0.92 | clear | | en Emma (B-) | 0.99 | 0.92 | natural | | en Cori (Piper) | 0.92 | 0.67 | clear | | es Dora, Alex | 1.00, 0.99 | 1.00, 0.95 | clear | | fr Siwis (B-) | 0.98 | 0.92 | natural | | it Sara, Nicola (C) | 0.98, 0.96 | 0.89, 0.87 | clear | | nl Reader, Flemish (Piper) | 0.95 | 0.90 | clear | | zh Xiaobei, Yunjian (D) | 0.97, 0.96 | 0.77, 0.71 | clear | | pt Dora, Alex | 0.86, 0.84 | 0.40, 0.55 | experimental ("mãe", "não" garbled) | | hi Alpha, Omega (C) | 0.87, 0.88 | 0.62, 0.69 | experimental ("नहीं" garbled) | | ja Alpha, Kumo (C+, C-) | 0.69, 0.63 | 0.29 | withheld: misreads 外, 好き, 疲れた | | nl Reader, Netherlands (Piper MLS) | 0.27 | 0.00 | withheld: babble | | de Reader (Piper MLS) | 0.14 | 0.00 | withheld: babble | | uk Lada (Piper) | 0.18 | 0.00 | withheld: other words | The failures are the voices with this runtime, not the packaging: the same conversion makes Cori and the Flemish voice clear, Moonshine has its own Ukrainian pronunciation rules (`uk` is a rule-based locale in 0.1.5), and both Japanese voices misread the same words, so its Japanese reading is at fault. Test a withheld voice again after a runtime update by deleting its entry in `intelligibility.json`: it is then built, rated experimental, and measured. A voice withheld this way, or by hand (`withheld` on its definition, as for Norman), is not built or offered. Its reason goes into the catalogue (the language's `withheld` list, which the app ignores; a language left with no voice says why, in plain words, under "Why") and into `LICENSES.md`. It keeps its definition and its pins. ## Samples Every offered voice has a sample: two or three seconds of it saying its language's `SAMPLE_TEXT` (a greeting and a request), MP3 at 32 kbit/s, about 10 KB. The app plays it, with the sentence written beside it, before the person downloads the voice, so nobody spends 100 MB on a voice they would not choose. `render_samples.mjs` renders them from the built packs with LAME (peaks kept at -1 dBFS) into `samples/` with `samples/index.json`; they are committed, because synthesis is not repeatable (Piper samples its durations), and `build_voices.py` publishes each under `samples//` beside the voices, only while it says the current text. Listen to every sample before committing it. A sample is the voice as it is: Portuguese Dora's says "mais açúcar" (sugar) for "mais suco", and that is what the person should hear. A voice from an anonymous dataset speaker is called "Reader" (with the region, where a language has two), never by its dataset id. ## Running it ```bash cd aac/aac-rs python3.14 tools/voices/build_voices.py # build and verify every pin (~15 s once cached) node tools/voices/smoke.mjs # synthesize with every pack (~35 s, Node >= 22) node tools/voices/smoke.mjs --wav /tmp/v en # also write WAVs to listen to python3 tools/voices/check_packs.py ../../../aac-voice-packs # what every site build runs first node tools/voices/render_samples.mjs # samples missing or stale, then listen and commit python tools/voices/intelligibility.py # measure (MLX Python with mlx-whisper; Apple Silicon) ``` The packs go to `aac-voice-packs` beside the open-standards checkout (`$AAC_VOICES_DIR` overrides it, and `--out` overrides both): the directory the site builds copy from. The pointer `index.json` is written last, after every file it names. What you need: - Python >= 3.11, standard library only. - For the Piper conversion, python3.12 or python3.13 on PATH, because onnxruntime 1.23.2 has no 3.14 wheels. On first use the builder creates `.cache/ort-venv/` and installs `ORT_PINS` into it as wheels. - Network access the first time, about 1 GB. After that, `--offline` builds from the cache alone. ### Flags - `--update-lock` records a pin for any source that is not yet in `sources.lock.json`. Without it, an unpinned source stops the build. - `--worker PATH` ships a specific `voice-worker.js`. By default the builder takes `third_party/moonshine-wasm/voice-worker.js` if it exists. - `--out DIR` writes somewhere other than the directory the site builds read. - `--converter-python` sets which Python builds the converter venv. ## Pins and reproducibility - **What is pinned, and how:** - GitHub and Hugging Face files by commit. Every revision is a constant at the top of `build_voices.py`, read on 2026-09-29. - The npm tarball by version and by the registry's sha512 integrity. - Every downloaded file by exact size and SHA-256 in `sources.lock.json`. - Piper `.onnx` files are also pinned in the script to their Hugging Face LFS oid. - **The Moonshine CDN has no revisions.** Files from `download.moonshine.ai/tts/` are pinned by content only. If Moonshine replaces a file, the build stops with a checksum mismatch rather than shipping something different. - **Downloads are cached.** They go once into `.cache/src/` (git-ignored) and are re-hashed on every run. - **Rebuilds are byte-identical.** A second run gives the same bytes and the same mtimes: every output file is stamped with `RETRIEVED`. This was checked with a fresh conversion, after deleting `.cache/conv`, and with an `--offline` run. The tree hash matched every time. - **The ORT conversion needs one workaround.** `convert_onnx_models_to_ort` writes the kernel-type table in the order of an absl hash map, and absl seeds that hash from an address. Two ordinary runs therefore differ in about 3 KB near the start of the file. The builder runs the converter with ASLR off (on macOS a `posix_spawn` flag, on Linux `setarch -R`), which makes the output stable. The same bytes came out of two different venvs. The converted SHA-256 is pinned in the lock's `converted` table. A different onnxruntime wheel on another platform may still produce other bytes; the build then stops, and after checking the voice with `smoke.mjs` you re-pin with `--update-lock`. - **To change a pin:** edit the constant, bump `RETRIEVED`, run with `--update-lock`, review the diff of `sources.lock.json`, and commit it together with `data/voices/`. - **Everything is the latest stable release, with one deliberate match.** Moonshine WASM 0.1.5 is the newest on npm (checked 2026-09-30). The Piper converter is onnxruntime 1.23.2, not the newest (1.30.0), because its ORT-format output has to load in the ORT 1.23 core linked into `moonshine.wasm`. It moves when a Moonshine release links a newer ORT. ## Output: the pack format Everything in the output directory is served at `/app/voices/` (and at `/voices/` on aac.transaction.science): | path | what | |---|---| | `index.json` | pointer `{schema, current}`; served `no-cache`, fetched `no-store` | | `index..json` | the catalogue (same bytes as the committed `data/voices/index.json`) | | `LICENSES.md` | attribution for everything; served `no-cache` (it is not content-addressed) | | `runtime/moonshine-wasm-0.1.5-/` | `moonshine.mjs`, `moonshine.wasm`, `voice-worker.js` | | `c/kokoro/..` | Kokoro-82M core (prosody and decoder stages, config), 83.4 MB | | `c/g2p//..` | Moonshine G2P data (en_us, fr, de, nl, hi, it, pt_br, ja, zh_hans) | | `v//..` | `.kokorovoice`, or a Piper `.ort` plus its `.onnx.json` | | `licences/..txt` | full licence text, upstream notices and sources for each runtime, component and pack | Every name except `index.json` and `LICENSES.md` carries the first 12 hex digits of its SHA-256, or sits in a content-addressed directory. Those names match `sw.js` `IMMUTABLE`, and a changed file always gets a new URL. `FileRef.name` is the key the file is handed to `TextToSpeech` under (`kokoro/prosody.model.ort`, `en_us/oov/model.ort`, `piper/onnx`, `piper/onnx.json`, ...). `smoke.mjs` checks that these keys cover the runtime's own `ttsDependencies()` list for every pack. ### Where this differs from the plan, and why 1. **Runtime files keep their npm names, inside a content-addressed directory.** Emscripten starts its pthreads with `new Worker(new URL("moonshine.mjs", import.meta.url))`, so the file must be called `moonshine.mjs`, and renaming it would break threading. 2. **Only `moonshine.mjs`, `moonshine.wasm` and `voice-worker.js` are shipped.** The worker uses the low-level module API. The package's TypeScript layer is unused, and it defaults to fetching from `download.moonshine.ai`, which the app must never do. 3. **No `voice`-kind components.** Each pack carries its own voice files in `files`, and its licence in `licence`, `attribution` and `licence_text`. The only components are `kokoro-core@1` and `g2p-@1`. 4. **Piper options are `{"vocoder_engine":"piper","voice":"piper_"}`.** With `vocoder_engine` alone, en_us still looks for `kokoro/config.json`, because its default voice is Kokoro. The `voice` key is harmless for the other languages. 5. **Extra fields**, all additive, which serde ignores: - top level: `base`; - runtime: `dir`, `module`, `wasm`, `requires`, `total_bytes`, `licence_text` and `sources`; - components: `total_bytes`, `licence_text`, `sources` and `note`; - packs: `total_bytes_with_runtime`, `licence_text` and `sources`. `reason` is left out for neural languages rather than set to null, because `neural_voice.rs` reads it as a `String`. 6. **How `min_ai_tier` maps to `AiTier::as_attr`:** - Kokoro is `wasm-only`, so it runs on any device above Minimal. Its peak RSS is about 0.5 GB, and it never uses WebGPU, so gating it on a WebGPU tier would exclude devices for no reason. - Piper is `minimal`. 7. **Added `en-piper-cori`** (en_GB-cori-high, LibriVox public domain, trained from scratch, MIT repository). The plan named it as Norman's clean alternative, and it is clean on the single words where Norman fails (see below). 8. **`data/voices/index.json` is the catalogue, not the pointer.** The pointer alone would be useless in git. aac-aurasense-web's `sync-languages.mjs` reads it, so /languages/ cannot disagree with the app, and `check_packs.py` holds the built packs to it. ## Sizes These are download sizes in MB. The runtime (13.3 MB) is fetched once. Shared components are counted once per device, so after the first Kokoro language, the next one costs only its G2P data and 0.5 MB of voice. | pack | own | total | + runtime | |---|---:|---:|---:| | en-kokoro-af_heart (default) | 0.5 | 109.0 | 122.3 | | en-kokoro-am_michael / bf_emma | 0.5 | 109.0 | 122.3 | | en-piper-cori | 114.9 | 140.0 | 153.3 | | es-kokoro-ef_dora (default) / em_alex | 0.5 | 83.9 | 97.3 | | fr-kokoro-ff_siwis (default) | 0.5 | 115.4 | 128.7 | | pt-kokoro-pf_dora (default) / pm_alex | 0.5 | 86.6 | 99.9 | | it-kokoro-if_sara (default) / im_nicola | 0.5 | 85.7 | 99.0 | | nl-piper-rdh (default) | 63.7 | 67.4 | 80.8 | | zh-kokoro-zf_xiaobei (default) / zm_yunjian | 0.5 | 187.8 | 201.1 | | hi-kokoro-hf_alpha (default) / hm_omega | 0.5 | 84.0 | 97.3 | Each pack's sample adds about 10 KB, downloaded only when played. The whole output is 93 files and 452 MB. The largest single file is Cori's Piper model, about 115 MB (the Chinese RoBERTa weights are next, about 102 MB), under model_store's 256 MB limit. ## Smoke test results (Node 26, MacBook Pro M5 Max, 2026-09-29) All 21 packs built then (the 16 offered now, with de, ja, uk and nl's MLS voice; 22 with Norman, before it was withheld) loaded from the output directory alone and synthesized a request and a single word at 24 kHz. That showed they run, not that they say the words: `intelligibility.py` came later, and withheld five of them. | engine | load | real-time factor | |---|---|---| | Kokoro | 0.2–0.9 s | about 0.35–0.47 | | Piper | 0.4–1.9 s | about 0.03–0.22 | Node's RSS peaked at about 2.4 GB with all the runtimes and packs loaded one after another in one process. These are desktop numbers: measure on an iPad and a mid-range Android before choosing defaults. Quality problems found, and recorded in each voice's `quality`: - **en-piper-norman (withheld):** the single word "more" comes out as 3–5 s of babble. It does the same with a plain onnxruntime reference inference of the upstream `.onnx`, so the fault is in the voice, not in Moonshine or the conversion. "juice" is fine. `en-piper-cori` is the light English voice instead. - **de-piper-mls (withheld):** speaker 0 takes 5–7 s over "Ich möchte mehr Saft, bitte." (4.9 s and 7.0 s in two runs on 2026-09-30), also under reference inference, where about 2 s is a normal pace; the app's speech rate (the runtime's `speed`) shortens it only to 3.7 s at 2.0, the fastest. It was first offered as experimental on the belief that it said the right words slowly. Measured, it does not: a recognizer understood 14% of its everyday phrases. Moonshine 0.1.5 hard-codes `sid = 0`, so no other MLS speaker can be tried; the Dutch MLS voice fails the same way. - **Sentences are the runtime's own.** The app splits a message where Moonshine's `ttsSplitUtterances` does (the utterances its `say()` makes), so "1.5 mg", "1.000 Euro" and "Dr. Smith" stay whole. It does take a German ordinal for the end of a sentence: "Am 3. Oktober" is said as "Am 3." and "Oktober", by `say()` as much as by the app. `smoke.mjs` flags a single word that takes longer than 2.5 s as `?`. That is a warning, not a failure. ## Deploying The packs ride along with each site's own build and deploy; nothing is copied by hand. 1. Build the packs on dev (into `aac-voice-packs` beside the open-standards checkout, by default), commit `data/voices/`, then run `smoke.mjs`: ```bash python3.14 tools/voices/build_voices.py node tools/voices/smoke.mjs ``` 2. The site builds check them with `check_packs.py` against the committed catalogue, then copy them in (cloned, not copied, on APFS), each from `$AAC_VOICES_DIR` when set: aac-aurasense-web's `scripts/build-app.sh` into `dist/app/voices/` (served at `aac.aurasense.ai/app/voices/`), and `prerender-static.sh` into `voices/` (served at `aac.transaction.science/voices/`). Packs that are missing, stale or incomplete stop the build; `AAC_VOICES=none` builds without voices on purpose (CI does), and the site then offers none. 3. aac-aurasense-web's `scripts/precompress.mjs` writes brotli and gzip sidecars for the pronunciation tables and the Kokoro and Chinese models (a French voice downloads as about 78 MB instead of 129); the sidecars are cached by content, so only new files cost time. 4. Caddy (site-ops `Caddyfile`, the `aac.aurasense.ai` and `aac.transaction.science` blocks) sends: - `Cache-Control: public, max-age=31536000, immutable, no-transform` for every content-addressed file (a hash in its name, or in the runtime's directory name), and `no-cache, no-transform` for `index.json` and `LICENSES.md`, which the app also fetches with `cache: "no-store"`; - COOP `same-origin` and COEP `require-corp` (everything is same-origin, so no CORP header is needed); - `.wasm` as `application/wasm` and `.mjs` as `text/javascript`. The other files are fetched as bytes and checked against their SHA-256, so their type does not matter; - a real 404 for a missing file, never the app's index page. 5. After a deploy, check that the pointer names a catalogue that is served, on each host: ```bash B=https://aac.aurasense.ai/app/voices curl -fsS "$B/$(curl -fsS "$B/index.json" | python3 -c 'import json,sys; print(json.load(sys.stdin)["current"])')" | head -c 80 ``` The site builds give `index.json` and `LICENSES.md` the build's own modification time, because the builder stamps every file with one fixed time and rsync compares size and time: otherwise a new pointer of the same length would not be copied. 6. The app vendors the same runtime separately, through `third_party/moonshine-wasm/vendor.sh` into `/app/vendor/`. This build's `runtime/` directory is the copy the pack format describes, and its licence entry is the one the Licences screen shows. Serve both, or drop one once `neural_voice.rs` settles on a single location. ## Licences `LICENSES.md` lists every runtime, component and pack, with its licence, attribution, sources and size. Each `licences/*.txt` file holds the upstream notices verbatim, fetched at pinned commits, with the full SPDX texts from `spdx/license-list-data` v3.29.0. Nothing is NC or GPL. Obligations the app has to meet: - **Label every neural voice "synthetic voice".** This covers the StyleTTS 2 pre-trained model condition that Kokoro inherits. - **Show every attribution on a Licences / Sources screen.** EDRDG, whose EDICT data the ja G2P uses, requires the acknowledgement on a separate About or Sources screen, not only a launch page. - **Offer the MPL-2.0 source.** Eigen is linked into `moonshine.wasm` under MPL-2.0, whose source is https://gitlab.com/libeigen/eigen. - **Keep ShareAlike files under their own licence.** These files are CC BY-SA: - the de and hi dictionaries; - the it WikiPron data; - the ja EDICT data and the ja RoBERTa model. If any of them is modified, the modified version stays CC BY-SA. Licence evidence was read on 2026-09-29. The URL for each item is recorded in the catalogue (`licence_url`, `sources`) and in its licence text.