open-standards/aac/aac-rs/tools/voices

Neural voice packs

build_voices.py builds the optional on-device neural voices for the AAC Reference App: one runtime (Moonshine WASM 0.1.5), the shared Kokoro-82M core, the G2P data for each language, and the voices themselves. It writes them in the aac-voice-index/1 pack format with a catalogue and full licence texts. Neither the pack files nor the app bundle go into git. The catalogue and LICENSES.md are committed in data/voices/; each voice's sample and the measurements its rating rests on are committed here.

v1 offers 16 voices in 8 of the 45 languages: en, es, fr, pt, it, zh and hi on Kokoro, and nl (Flemish) on Piper. ha and yo are planned for v1.1 (OpenBible VITS on onnxruntime-web). de, ja and uk had voices, which are withheld because they say other words (below); with the other 32 languages, they are web-speech-only, each with its reason in the catalogue and in LICENSES.md.

Ratings, and voices not offered

A voice must say the words the person chose. That is measured, not assumed: intelligibility.py has every voice say its language's seven everyday phrases (phrases.json: a request, help, stop, yes, no, a feeling, a question), a speech recognizer (Whisper large-v3-turbo) writes down what it hears, and the score is how much of each phrase it got back (0 to 1, ignoring case, spaces and punctuation). The results are committed in intelligibility.json, and build_voices.py derives each voice's rating from them and from the upstream naturalness grade (Kokoro's VOICES.md), by RATING_RULE:

rating	earned by	the app says
withheld	under 0.80 on average: it says other words	(not offered)
experimental	under 0.90 on average, any phrase under 0.60, or never measured	"Experimental: it may not sound right. Listen to it before relying on it."
clear	0.90 and over	"Clear and easy to understand, though less natural."
natural	0.95 and over, with an upstream grade of A or B	"Sounds natural."

A missing or unknown rating reads as experimental in the app. Each pack carries its intelligibility (mean, min, model, date) for maintainers, beside quality, the notes behind the grade; neither is shown.

Measured on 2026-09-30, twice (Piper voices vary a little from run to run):

voice	mean	lowest	rating
en Heart (A)	1.00	1.00	natural
en Michael (C+)	0.99	0.92	clear
en Emma (B-)	0.99	0.92	natural
en Cori (Piper)	0.92	0.67	clear
es Dora, Alex	1.00, 0.99	1.00, 0.95	clear
fr Siwis (B-)	0.98	0.92	natural
it Sara, Nicola (C)	0.98, 0.96	0.89, 0.87	clear
nl Reader, Flemish (Piper)	0.95	0.90	clear
zh Xiaobei, Yunjian (D)	0.97, 0.96	0.77, 0.71	clear
pt Dora, Alex	0.86, 0.84	0.40, 0.55	experimental ("mãe", "não" garbled)
hi Alpha, Omega (C)	0.87, 0.88	0.62, 0.69	experimental ("नहीं" garbled)
ja Alpha, Kumo (C+, C-)	0.69, 0.63	0.29	withheld: misreads 外, 好き, 疲れた
nl Reader, Netherlands (Piper MLS)	0.27	0.00	withheld: babble
de Reader (Piper MLS)	0.14	0.00	withheld: babble
uk Lada (Piper)	0.18	0.00	withheld: other words

The failures are the voices with this runtime, not the packaging: the same conversion makes Cori and the Flemish voice clear, Moonshine has its own Ukrainian pronunciation rules (uk is a rule-based locale in 0.1.5), and both Japanese voices misread the same words, so its Japanese reading is at fault. Test a withheld voice again after a runtime update by deleting its entry in intelligibility.json: it is then built, rated experimental, and measured.

A voice withheld this way, or by hand (withheld on its definition, as for Norman), is not built or offered. Its reason goes into the catalogue (the language's withheld list, which the app ignores; a language left with no voice says why, in plain words, under "Why") and into LICENSES.md. It keeps its definition and its pins.

Samples

Every offered voice has a sample: two or three seconds of it saying its language's SAMPLE_TEXT (a greeting and a request), MP3 at 32 kbit/s, about 10 KB. The app plays it, with the sentence written beside it, before the person downloads the voice, so nobody spends 100 MB on a voice they would not choose. render_samples.mjs renders them from the built packs with LAME (peaks kept at -1 dBFS) into samples/ with samples/index.json; they are committed, because synthesis is not repeatable (Piper samples its durations), and build_voices.py publishes each under samples/<lang>/ beside the voices, only while it says the current text. Listen to every sample before committing it. A sample is the voice as it is: Portuguese Dora's says "mais açúcar" (sugar) for "mais suco", and that is what the person should hear.

A voice from an anonymous dataset speaker is called "Reader" (with the region, where a language has two), never by its dataset id.

Running it

cd aac/aac-rs
python3.14 tools/voices/build_voices.py        # build and verify every pin (~15 s once cached)
node tools/voices/smoke.mjs                    # synthesize with every pack (~35 s, Node >= 22)
node tools/voices/smoke.mjs --wav /tmp/v en    # also write WAVs to listen to
python3 tools/voices/check_packs.py ../../../aac-voice-packs   # what every site build runs first
node tools/voices/render_samples.mjs           # samples missing or stale, then listen and commit
python tools/voices/intelligibility.py         # measure (MLX Python with mlx-whisper; Apple Silicon)

The packs go to aac-voice-packs beside the open-standards checkout ($AAC_VOICES_DIR overrides it, and --out overrides both): the directory the site builds copy from. The pointer index.json is written last, after every file it names.

What you need:

• Python >= 3.11, standard library only.
• For the Piper conversion, python3.12 or python3.13 on PATH, because onnxruntime 1.23.2 has no 3.14 wheels. On first use the builder creates .cache/ort-venv/ and installs ORT_PINS into it as wheels.
• Network access the first time, about 1 GB. After that, --offline builds from the cache alone.

Flags

• --update-lock records a pin for any source that is not yet in sources.lock.json. Without it, an unpinned source stops the build.
• --worker PATH ships a specific voice-worker.js. By default the builder takes third_party/moonshine-wasm/voice-worker.js if it exists.
• --out DIR writes somewhere other than the directory the site builds read.
• --converter-python sets which Python builds the converter venv.

Pins and reproducibility

• What is pinned, and how:
  • GitHub and Hugging Face files by commit. Every revision is a constant at the top of build_voices.py, read on 2026-09-29.
  • The npm tarball by version and by the registry's sha512 integrity.
  • Every downloaded file by exact size and SHA-256 in sources.lock.json.
  • Piper .onnx files are also pinned in the script to their Hugging Face LFS oid.
• The Moonshine CDN has no revisions. Files from download.moonshine.ai/tts/ are pinned by content only. If Moonshine replaces a file, the build stops with a checksum mismatch rather than shipping something different.
• Downloads are cached. They go once into .cache/src/<sha256> (git-ignored) and are re-hashed on every run.
• Rebuilds are byte-identical. A second run gives the same bytes and the same mtimes: every output file is stamped with RETRIEVED. This was checked with a fresh conversion, after deleting .cache/conv, and with an --offline run. The tree hash matched every time.
• The ORT conversion needs one workaround. convert_onnx_models_to_ort writes the kernel-type table in the order of an absl hash map, and absl seeds that hash from an address. Two ordinary runs therefore differ in about 3 KB near the start of the file. The builder runs the converter with ASLR off (on macOS a posix_spawn flag, on Linux setarch -R), which makes the output stable. The same bytes came out of two different venvs. The converted SHA-256 is pinned in the lock's converted table. A different onnxruntime wheel on another platform may still produce other bytes; the build then stops, and after checking the voice with smoke.mjs you re-pin with --update-lock.
• To change a pin: edit the constant, bump RETRIEVED, run with --update-lock, review the diff of sources.lock.json, and commit it together with data/voices/.
• Everything is the latest stable release, with one deliberate match. Moonshine WASM 0.1.5 is the newest on npm (checked 2026-09-30). The Piper converter is onnxruntime 1.23.2, not the newest (1.30.0), because its ORT-format output has to load in the ORT 1.23 core linked into moonshine.wasm. It moves when a Moonshine release links a newer ORT.

Output: the pack format

Everything in the output directory is served at /app/voices/ (and at /voices/ on aac.transaction.science):

path	what
index.json	pointer {schema, current}; served no-cache, fetched no-store
index.<sha12>.json	the catalogue (same bytes as the committed data/voices/index.json)
LICENSES.md	attribution for everything; served no-cache (it is not content-addressed)
runtime/moonshine-wasm-0.1.5-<sha12>/	moonshine.mjs, moonshine.wasm, voice-worker.js
c/kokoro/<file>.<sha12>.<ext>	Kokoro-82M core (prosody and decoder stages, config), 83.4 MB
c/g2p/<lang>/<path>.<sha12>.<ext>	Moonshine G2P data (en_us, fr, de, nl, hi, it, pt_br, ja, zh_hans)
v/<lang>/<voice>.<sha12>.<ext>	.kokorovoice, or a Piper .ort plus its .onnx.json
licences/<id>.<sha12>.txt	full licence text, upstream notices and sources for each runtime, component and pack

Every name except index.json and LICENSES.md carries the first 12 hex digits of its SHA-256, or sits in a content-addressed directory. Those names match sw.js IMMUTABLE, and a changed file always gets a new URL. FileRef.name is the key the file is handed to TextToSpeech under (kokoro/prosody.model.ort, en_us/oov/model.ort, piper/onnx, piper/onnx.json, ...). smoke.mjs checks that these keys cover the runtime's own ttsDependencies() list for every pack.

Where this differs from the plan, and why

1. Runtime files keep their npm names, inside a content-addressed directory. Emscripten starts its pthreads with new Worker(new URL("moonshine.mjs", import.meta.url)), so the file must be called moonshine.mjs, and renaming it would break threading.

2. Only moonshine.mjs, moonshine.wasm and voice-worker.js are shipped. The worker uses the low-level module API. The package's TypeScript layer is unused, and it defaults to fetching from download.moonshine.ai, which the app must never do.

3. No voice-kind components. Each pack carries its own voice files in files, and its licence in licence, attribution and licence_text. The only components are kokoro-core@1 and g2p-<lang>@1.

4. Piper options are {"vocoder_engine":"piper","voice":"piper_<stem>"}. With vocoder_engine alone, en_us still looks for kokoro/config.json, because its default voice is Kokoro. The voice key is harmless for the other languages.

5. Extra fields, all additive, which serde ignores:

   • top level: base;
   • runtime: dir, module, wasm, requires, total_bytes, licence_text and sources;
   • components: total_bytes, licence_text, sources and note;
   • packs: total_bytes_with_runtime, licence_text and sources.

   reason is left out for neural languages rather than set to null, because neural_voice.rs reads it as a String.

6. How min_ai_tier maps to AiTier::as_attr:

   • Kokoro is wasm-only, so it runs on any device above Minimal. Its peak RSS is about 0.5 GB, and it never uses WebGPU, so gating it on a WebGPU tier would exclude devices for no reason.
   • Piper is minimal.

7. Added en-piper-cori (en_GB-cori-high, LibriVox public domain, trained from scratch, MIT repository). The plan named it as Norman's clean alternative, and it is clean on the single words where Norman fails (see below).

8. data/voices/index.json is the catalogue, not the pointer. The pointer alone would be useless in git. aac-aurasense-web's sync-languages.mjs reads it, so /languages/ cannot disagree with the app, and check_packs.py holds the built packs to it.

Sizes

These are download sizes in MB. The runtime (13.3 MB) is fetched once. Shared components are counted once per device, so after the first Kokoro language, the next one costs only its G2P data and 0.5 MB of voice.

pack	own	total	+ runtime
en-kokoro-af_heart (default)	0.5	109.0	122.3
en-kokoro-am_michael / bf_emma	0.5	109.0	122.3
en-piper-cori	114.9	140.0	153.3
es-kokoro-ef_dora (default) / em_alex	0.5	83.9	97.3
fr-kokoro-ff_siwis (default)	0.5	115.4	128.7
pt-kokoro-pf_dora (default) / pm_alex	0.5	86.6	99.9
it-kokoro-if_sara (default) / im_nicola	0.5	85.7	99.0
nl-piper-rdh (default)	63.7	67.4	80.8
zh-kokoro-zf_xiaobei (default) / zm_yunjian	0.5	187.8	201.1
hi-kokoro-hf_alpha (default) / hm_omega	0.5	84.0	97.3

Each pack's sample adds about 10 KB, downloaded only when played. The whole output is 93 files and 452 MB. The largest single file is Cori's Piper model, about 115 MB (the Chinese RoBERTa weights are next, about 102 MB), under model_store's 256 MB limit.

Smoke test results (Node 26, MacBook Pro M5 Max, 2026-09-29)

All 21 packs built then (the 16 offered now, with de, ja, uk and nl's MLS voice; 22 with Norman, before it was withheld) loaded from the output directory alone and synthesized a request and a single word at 24 kHz. That showed they run, not that they say the words: intelligibility.py came later, and withheld five of them.

engine	load	real-time factor
Kokoro	0.2–0.9 s	about 0.35–0.47
Piper	0.4–1.9 s	about 0.03–0.22

Node's RSS peaked at about 2.4 GB with all the runtimes and packs loaded one after another in one process. These are desktop numbers: measure on an iPad and a mid-range Android before choosing defaults.

Quality problems found, and recorded in each voice's quality:

• en-piper-norman (withheld): the single word "more" comes out as 3–5 s of babble. It does the same with a plain onnxruntime reference inference of the upstream .onnx, so the fault is in the voice, not in Moonshine or the conversion. "juice" is fine. en-piper-cori is the light English voice instead.

• de-piper-mls (withheld): speaker 0 takes 5–7 s over "Ich möchte mehr Saft, bitte." (4.9 s and 7.0 s in two runs on 2026-09-30), also under reference inference, where about 2 s is a normal pace; the app's speech rate (the runtime's speed) shortens it only to 3.7 s at 2.0, the fastest. It was first offered as experimental on the belief that it said the right words slowly. Measured, it does not: a recognizer understood 14% of its everyday phrases. Moonshine 0.1.5 hard-codes sid = 0, so no other MLS speaker can be tried; the Dutch MLS voice fails the same way.

• Sentences are the runtime's own. The app splits a message where Moonshine's ttsSplitUtterances does (the utterances its say() makes), so "1.5 mg", "1.000 Euro" and "Dr. Smith" stay whole. It does take a German ordinal for the end of a sentence: "Am 3. Oktober" is said as "Am 3." and "Oktober", by say() as much as by the app.

smoke.mjs flags a single word that takes longer than 2.5 s as ?. That is a warning, not a failure.

Deploying

The packs ride along with each site's own build and deploy; nothing is copied by hand.

1. Build the packs on dev (into aac-voice-packs beside the open-standards checkout, by default), commit data/voices/, then run smoke.mjs:

   python3.14 tools/voices/build_voices.py
   node tools/voices/smoke.mjs
   

2. The site builds check them with check_packs.py against the committed catalogue, then copy them in (cloned, not copied, on APFS), each from $AAC_VOICES_DIR when set: aac-aurasense-web's scripts/build-app.sh into dist/app/voices/ (served at aac.aurasense.ai/app/voices/), and prerender-static.sh into voices/ (served at aac.transaction.science/voices/). Packs that are missing, stale or incomplete stop the build; AAC_VOICES=none builds without voices on purpose (CI does), and the site then offers none.
3. aac-aurasense-web's scripts/precompress.mjs writes brotli and gzip sidecars for the pronunciation tables and the Kokoro and Chinese models (a French voice downloads as about 78 MB instead of 129); the sidecars are cached by content, so only new files cost time.
4. Caddy (site-ops Caddyfile, the aac.aurasense.ai and aac.transaction.science blocks) sends:
   • Cache-Control: public, max-age=31536000, immutable, no-transform for every content-addressed file (a hash in its name, or in the runtime's directory name), and no-cache, no-transform for index.json and LICENSES.md, which the app also fetches with cache: "no-store";
   • COOP same-origin and COEP require-corp (everything is same-origin, so no CORP header is needed);
   • .wasm as application/wasm and .mjs as text/javascript. The other files are fetched as bytes and checked against their SHA-256, so their type does not matter;
   • a real 404 for a missing file, never the app's index page.
5. After a deploy, check that the pointer names a catalogue that is served, on each host:

   B=https://aac.aurasense.ai/app/voices
   curl -fsS "$B/$(curl -fsS "$B/index.json" | python3 -c 'import json,sys; print(json.load(sys.stdin)["current"])')" | head -c 80
   

   The site builds give index.json and LICENSES.md the build's own modification time, because the builder stamps every file with one fixed time and rsync compares size and time: otherwise a new pointer of the same length would not be copied.
6. The app vendors the same runtime separately, through third_party/moonshine-wasm/vendor.sh into /app/vendor/. This build's runtime/ directory is the copy the pack format describes, and its licence entry is the one the Licences screen shows. Serve both, or drop one once neural_voice.rs settles on a single location.

Licences

LICENSES.md lists every runtime, component and pack, with its licence, attribution, sources and size. Each licences/*.txt file holds the upstream notices verbatim, fetched at pinned commits, with the full SPDX texts from spdx/license-list-data v3.29.0. Nothing is NC or GPL.

Obligations the app has to meet:

• Label every neural voice "synthetic voice". This covers the StyleTTS 2 pre-trained model condition that Kokoro inherits.

• Show every attribution on a Licences / Sources screen. EDRDG, whose EDICT data the ja G2P uses, requires the acknowledgement on a separate About or Sources screen, not only a launch page.

• Offer the MPL-2.0 source. Eigen is linked into moonshine.wasm under MPL-2.0, whose source is https://gitlab.com/libeigen/eigen.

• Keep ShareAlike files under their own licence. These files are CC BY-SA:

  • the de and hi dictionaries;
  • the it WikiPron data;
  • the ja EDICT data and the ja RoBERTa model.

  If any of them is modified, the modified version stays CC BY-SA.

Licence evidence was read on 2026-09-29. The URL for each item is recorded in the catalogue (licence_url, sources) and in its licence text.