open-standards/aac/aac-rs/tools/voices/README.md

# Neural voice packs

`build_voices.py` builds the optional on-device neural voices for the AAC
Reference App: one runtime (Moonshine WASM 0.1.5), the shared Kokoro-82M core,
the G2P data for each language, and the voices themselves. It writes them in
the `aac-voice-index/1` pack format with a catalogue and full licence texts.
Neither the pack files nor the app bundle go into git. The catalogue and
`LICENSES.md` are committed in `data/voices/`; each voice's sample and the
measurements its rating rests on are committed here.

v1 offers 16 voices in 8 of the 45 languages: en, es, fr, pt, it, zh and hi
on Kokoro, and nl (Flemish) on Piper. ha and yo are `planned` for v1.1
(OpenBible VITS on onnxruntime-web). de, ja and uk had voices, which are
withheld because they say other words (below); with the other 32 languages,
they are `web-speech-only`, each with its reason in the catalogue and in
`LICENSES.md`.

## Ratings, and voices not offered

A voice must say the words the person chose. That is measured, not assumed:
`intelligibility.py` has every voice say its language's seven everyday phrases
(`phrases.json`: a request, help, stop, yes, no, a feeling, a question), a
speech recognizer (Whisper large-v3-turbo) writes down what it hears, and the
score is how much of each phrase it got back (0 to 1, ignoring case, spaces
and punctuation). The results are committed in `intelligibility.json`, and
`build_voices.py` derives each voice's `rating` from them and from the
upstream naturalness grade (Kokoro's VOICES.md), by `RATING_RULE`:

| rating | earned by | the app says |
|---|---|---|
| withheld | under 0.80 on average: it says other words | (not offered) |
| `experimental` | under 0.90 on average, any phrase under 0.60, or never measured | "Experimental: it may not sound right. Listen to it before relying on it." |
| `clear` | 0.90 and over | "Clear and easy to understand, though less natural." |
| `natural` | 0.95 and over, with an upstream grade of A or B | "Sounds natural." |

A missing or unknown rating reads as `experimental` in the app. Each pack
carries its `intelligibility` (mean, min, model, date) for maintainers, beside
`quality`, the notes behind the grade; neither is shown.

Measured on 2026-09-30, twice (Piper voices vary a little from run to run):

| voice | mean | lowest | rating |
|---|---:|---:|---|
| en Heart (A) | 1.00 | 1.00 | natural |
| en Michael (C+) | 0.99 | 0.92 | clear |
| en Emma (B-) | 0.99 | 0.92 | natural |
| en Cori (Piper) | 0.92 | 0.67 | clear |
| es Dora, Alex | 1.00, 0.99 | 1.00, 0.95 | clear |
| fr Siwis (B-) | 0.98 | 0.92 | natural |
| it Sara, Nicola (C) | 0.98, 0.96 | 0.89, 0.87 | clear |
| nl Reader, Flemish (Piper) | 0.95 | 0.90 | clear |
| zh Xiaobei, Yunjian (D) | 0.97, 0.96 | 0.77, 0.71 | clear |
| pt Dora, Alex | 0.86, 0.84 | 0.40, 0.55 | experimental ("mãe", "não" garbled) |
| hi Alpha, Omega (C) | 0.87, 0.88 | 0.62, 0.69 | experimental ("नहीं" garbled) |
| ja Alpha, Kumo (C+, C-) | 0.69, 0.63 | 0.29 | withheld: misreads 外, 好き, 疲れた |
| nl Reader, Netherlands (Piper MLS) | 0.27 | 0.00 | withheld: babble |
| de Reader (Piper MLS) | 0.14 | 0.00 | withheld: babble |
| uk Lada (Piper) | 0.18 | 0.00 | withheld: other words |

The failures are the voices with this runtime, not the packaging: the same
conversion makes Cori and the Flemish voice clear, Moonshine has its own
Ukrainian pronunciation rules (`uk` is a rule-based locale in 0.1.5), and
both Japanese voices misread the same words, so its Japanese reading is at
fault. Test a withheld voice again after a runtime update by deleting its
entry in `intelligibility.json`: it is then built, rated experimental, and
measured.

A voice withheld this way, or by hand (`withheld` on its definition, as for
Norman), is not built or offered. Its reason goes into the catalogue (the
language's `withheld` list, which the app ignores; a language left with no
voice says why, in plain words, under "Why") and into `LICENSES.md`. It keeps
its definition and its pins.

## Samples

Every offered voice has a sample: two or three seconds of it saying its
language's `SAMPLE_TEXT` (a greeting and a request), MP3 at 32 kbit/s, about
10 KB. The app plays it, with the sentence written beside it, before the
person downloads the voice, so nobody spends 100 MB on a voice they would not
choose. `render_samples.mjs` renders them from the built packs with LAME
(peaks kept at -1 dBFS) into `samples/` with `samples/index.json`; they are
committed, because synthesis is not repeatable (Piper samples its durations),
and `build_voices.py` publishes each under `samples/<lang>/` beside the voices,
only while it says the current text. Listen to every sample before committing
it. A sample is the voice as it is: Portuguese Dora's says "mais açúcar"
(sugar) for "mais suco", and that is what the person should hear.

A voice from an anonymous dataset speaker is called "Reader" (with the
region, where a language has two), never by its dataset id.

## Running it

```bash
cd aac/aac-rs
python3.14 tools/voices/build_voices.py        # build and verify every pin (~15 s once cached)
node tools/voices/smoke.mjs                    # synthesize with every pack (~35 s, Node >= 22)
node tools/voices/smoke.mjs --wav /tmp/v en    # also write WAVs to listen to
python3 tools/voices/check_packs.py ../../../aac-voice-packs   # what every site build runs first
node tools/voices/render_samples.mjs           # samples missing or stale, then listen and commit
python tools/voices/intelligibility.py         # measure (MLX Python with mlx-whisper; Apple Silicon)
```

The packs go to `aac-voice-packs` beside the open-standards checkout
(`$AAC_VOICES_DIR` overrides it, and `--out` overrides both): the directory
the site builds copy from. The pointer `index.json` is written last, after
every file it names.

What you need:

- Python >= 3.11, standard library only.
- For the Piper conversion, python3.12 or python3.13 on PATH, because
  onnxruntime 1.23.2 has no 3.14 wheels. On first use the builder creates
  `.cache/ort-venv/` and installs `ORT_PINS` into it as wheels.
- Network access the first time, about 1 GB. After that, `--offline` builds
  from the cache alone.

### Flags

- `--update-lock` records a pin for any source that is not yet in
  `sources.lock.json`. Without it, an unpinned source stops the build.
- `--worker PATH` ships a specific `voice-worker.js`. By default the builder
  takes `third_party/moonshine-wasm/voice-worker.js` if it exists.
- `--out DIR` writes somewhere other than the directory the site builds read.
- `--converter-python` sets which Python builds the converter venv.

## Pins and reproducibility

- **What is pinned, and how:**
  - GitHub and Hugging Face files by commit. Every revision is a constant at
    the top of `build_voices.py`, read on 2026-09-29.
  - The npm tarball by version and by the registry's sha512 integrity.
  - Every downloaded file by exact size and SHA-256 in `sources.lock.json`.
  - Piper `.onnx` files are also pinned in the script to their Hugging Face
    LFS oid.
- **The Moonshine CDN has no revisions.** Files from `download.moonshine.ai/tts/`
  are pinned by content only. If Moonshine replaces a file, the build stops
  with a checksum mismatch rather than shipping something different.
- **Downloads are cached.** They go once into `.cache/src/<sha256>` (git-ignored)
  and are re-hashed on every run.
- **Rebuilds are byte-identical.** A second run gives the same bytes and the
  same mtimes: every output file is stamped with `RETRIEVED`. This was
  checked with a fresh conversion, after deleting `.cache/conv`, and with an
  `--offline` run. The tree hash matched every time.
- **The ORT conversion needs one workaround.** `convert_onnx_models_to_ort`
  writes the kernel-type table in the order of an absl hash map, and absl
  seeds that hash from an address. Two ordinary runs therefore differ in
  about 3 KB near the start of the file. The builder runs the converter with
  ASLR off (on macOS a `posix_spawn` flag, on Linux `setarch -R`), which makes
  the output stable. The same bytes came out of two different venvs. The
  converted SHA-256 is pinned in the lock's `converted` table. A different
  onnxruntime wheel on another platform may still produce other bytes; the
  build then stops, and after checking the voice with `smoke.mjs` you
  re-pin with `--update-lock`.
- **To change a pin:** edit the constant, bump `RETRIEVED`, run with
  `--update-lock`, review the diff of `sources.lock.json`, and commit it
  together with `data/voices/`.
- **Everything is the latest stable release, with one deliberate match.**
  Moonshine WASM 0.1.5 is the newest on npm (checked 2026-09-30). The Piper
  converter is onnxruntime 1.23.2, not the newest (1.30.0), because its
  ORT-format output has to load in the ORT 1.23 core linked into
  `moonshine.wasm`. It moves when a Moonshine release links a newer ORT.

## Output: the pack format

Everything in the output directory is served at `/app/voices/` (and at
`/voices/` on aac.transaction.science):

| path | what |
|---|---|
| `index.json` | pointer `{schema, current}`; served `no-cache`, fetched `no-store` |
| `index.<sha12>.json` | the catalogue (same bytes as the committed `data/voices/index.json`) |
| `LICENSES.md` | attribution for everything; served `no-cache` (it is not content-addressed) |
| `runtime/moonshine-wasm-0.1.5-<sha12>/` | `moonshine.mjs`, `moonshine.wasm`, `voice-worker.js` |
| `c/kokoro/<file>.<sha12>.<ext>` | Kokoro-82M core (prosody and decoder stages, config), 83.4 MB |
| `c/g2p/<lang>/<path>.<sha12>.<ext>` | Moonshine G2P data (en_us, fr, de, nl, hi, it, pt_br, ja, zh_hans) |
| `v/<lang>/<voice>.<sha12>.<ext>` | `.kokorovoice`, or a Piper `.ort` plus its `.onnx.json` |
| `licences/<id>.<sha12>.txt` | full licence text, upstream notices and sources for each runtime, component and pack |

Every name except `index.json` and `LICENSES.md` carries the first 12 hex
digits of its SHA-256, or sits in a content-addressed directory. Those names
match `sw.js` `IMMUTABLE`, and a changed file always gets a new URL.
`FileRef.name` is the key the file is handed to `TextToSpeech` under
(`kokoro/prosody.model.ort`, `en_us/oov/model.ort`, `piper/onnx`,
`piper/onnx.json`, ...). `smoke.mjs` checks that these keys cover the
runtime's own `ttsDependencies()` list for every pack.

### Where this differs from the plan, and why

1. **Runtime files keep their npm names, inside a content-addressed directory.**
   Emscripten starts its pthreads with
   `new Worker(new URL("moonshine.mjs", import.meta.url))`, so the file must
   be called `moonshine.mjs`, and renaming it would break threading.
2. **Only `moonshine.mjs`, `moonshine.wasm` and `voice-worker.js` are shipped.**
   The worker uses the low-level module API. The package's TypeScript layer
   is unused, and it defaults to fetching from `download.moonshine.ai`, which
   the app must never do.
3. **No `voice`-kind components.** Each pack carries its own voice files in
   `files`, and its licence in `licence`, `attribution` and `licence_text`.
   The only components are `kokoro-core@1` and `g2p-<lang>@1`.
4. **Piper options are `{"vocoder_engine":"piper","voice":"piper_<stem>"}`.**
   With `vocoder_engine` alone, en_us still looks for `kokoro/config.json`,
   because its default voice is Kokoro. The `voice` key is harmless for the
   other languages.
5. **Extra fields**, all additive, which serde ignores:
   - top level: `base`;
   - runtime: `dir`, `module`, `wasm`, `requires`, `total_bytes`,
     `licence_text` and `sources`;
   - components: `total_bytes`, `licence_text`, `sources` and `note`;
   - packs: `total_bytes_with_runtime`, `licence_text` and `sources`.

   `reason` is left out for neural languages rather than set to null,
   because `neural_voice.rs` reads it as a `String`.
6. **How `min_ai_tier` maps to `AiTier::as_attr`:**
   - Kokoro is `wasm-only`, so it runs on any device above Minimal. Its
     peak RSS is about 0.5 GB, and it never uses WebGPU, so gating it on a
     WebGPU tier would exclude devices for no reason.
   - Piper is `minimal`.
7. **Added `en-piper-cori`** (en_GB-cori-high, LibriVox public domain, trained
   from scratch, MIT repository). The plan named it as Norman's clean
   alternative, and it is clean on the single words where Norman fails
   (see below).
8. **`data/voices/index.json` is the catalogue, not the pointer.** The
   pointer alone would be useless in git. aac-aurasense-web's
   `sync-languages.mjs` reads it, so /languages/ cannot disagree with the
   app, and `check_packs.py` holds the built packs to it.

## Sizes

These are download sizes in MB. The runtime (13.3 MB) is fetched once.
Shared components are counted once per device, so after the first Kokoro
language, the next one costs only its G2P data and 0.5 MB of voice.

| pack | own | total | + runtime |
|---|---:|---:|---:|
| en-kokoro-af_heart (default) | 0.5 | 109.0 | 122.3 |
| en-kokoro-am_michael / bf_emma | 0.5 | 109.0 | 122.3 |
| en-piper-cori | 114.9 | 140.0 | 153.3 |
| es-kokoro-ef_dora (default) / em_alex | 0.5 | 83.9 | 97.3 |
| fr-kokoro-ff_siwis (default) | 0.5 | 115.4 | 128.7 |
| pt-kokoro-pf_dora (default) / pm_alex | 0.5 | 86.6 | 99.9 |
| it-kokoro-if_sara (default) / im_nicola | 0.5 | 85.7 | 99.0 |
| nl-piper-rdh (default) | 63.7 | 67.4 | 80.8 |
| zh-kokoro-zf_xiaobei (default) / zm_yunjian | 0.5 | 187.8 | 201.1 |
| hi-kokoro-hf_alpha (default) / hm_omega | 0.5 | 84.0 | 97.3 |

Each pack's sample adds about 10 KB, downloaded only when played. The whole
output is 93 files and 452 MB. The largest single file is Cori's
Piper model, about 115 MB (the Chinese RoBERTa weights are next, about
102 MB), under model_store's 256 MB limit.

## Smoke test results (Node 26, MacBook Pro M5 Max, 2026-09-29)

All 21 packs built then (the 16 offered now, with de, ja, uk and nl's MLS
voice; 22 with Norman, before it was withheld) loaded from the output
directory alone and synthesized a request and a single word at 24 kHz. That
showed they run, not that they say the words: `intelligibility.py` came later,
and withheld five of them.

| engine | load | real-time factor |
|---|---|---|
| Kokoro | 0.2–0.9 s | about 0.35–0.47 |
| Piper | 0.4–1.9 s | about 0.03–0.22 |

Node's RSS peaked at about 2.4 GB with all the runtimes and packs loaded one
after another in one process. These are desktop numbers: measure on an iPad
and a mid-range Android before choosing defaults.

Quality problems found, and recorded in each voice's `quality`:

- **en-piper-norman (withheld):** the single word "more" comes out as 3–5 s
  of babble. It does the same with a plain onnxruntime reference inference of
  the upstream `.onnx`, so the fault is in the voice, not in Moonshine or the
  conversion. "juice" is fine. `en-piper-cori` is the light English voice
  instead.
- **de-piper-mls (withheld):** speaker 0 takes 5–7 s over "Ich möchte mehr
  Saft, bitte." (4.9 s and 7.0 s in two runs on 2026-09-30), also under
  reference inference, where about 2 s is a normal pace; the app's speech
  rate (the runtime's `speed`) shortens it only to 3.7 s at 2.0, the fastest.
  It was first offered as experimental on the belief that it said the right
  words slowly. Measured, it does not: a recognizer understood 14% of its
  everyday phrases. Moonshine 0.1.5 hard-codes `sid = 0`, so no other MLS
  speaker can be tried; the Dutch MLS voice fails the same way.

- **Sentences are the runtime's own.** The app splits a message where
  Moonshine's `ttsSplitUtterances` does (the utterances its `say()` makes), so
  "1.5 mg", "1.000 Euro" and "Dr. Smith" stay whole. It does take a German
  ordinal for the end of a sentence: "Am 3. Oktober" is said as "Am 3." and
  "Oktober", by `say()` as much as by the app.

`smoke.mjs` flags a single word that takes longer than 2.5 s as `?`. That is
a warning, not a failure.

## Deploying

The packs ride along with each site's own build and deploy; nothing is copied
by hand.

1. Build the packs on dev (into `aac-voice-packs` beside the open-standards
   checkout, by default), commit `data/voices/`, then run `smoke.mjs`:
   ```bash
   python3.14 tools/voices/build_voices.py
   node tools/voices/smoke.mjs
   ```
2. The site builds check them with `check_packs.py` against the committed
   catalogue, then copy them in (cloned, not copied, on APFS), each from
   `$AAC_VOICES_DIR` when set: aac-aurasense-web's `scripts/build-app.sh` into
   `dist/app/voices/` (served at `aac.aurasense.ai/app/voices/`), and
   `prerender-static.sh` into `voices/` (served at
   `aac.transaction.science/voices/`). Packs that are missing, stale or
   incomplete stop the build; `AAC_VOICES=none` builds without voices on
   purpose (CI does), and the site then offers none.
3. aac-aurasense-web's `scripts/precompress.mjs` writes brotli and gzip
   sidecars for the pronunciation tables and the Kokoro and Chinese models
   (a French voice downloads as about 78 MB instead of 129); the sidecars are
   cached by content, so only new files cost time.
4. Caddy (site-ops `Caddyfile`, the `aac.aurasense.ai` and
   `aac.transaction.science` blocks) sends:
   - `Cache-Control: public, max-age=31536000, immutable, no-transform` for
     every content-addressed file (a hash in its name, or in the runtime's
     directory name), and `no-cache, no-transform` for `index.json` and
     `LICENSES.md`, which the app also fetches with `cache: "no-store"`;
   - COOP `same-origin` and COEP `require-corp` (everything is same-origin,
     so no CORP header is needed);
   - `.wasm` as `application/wasm` and `.mjs` as `text/javascript`. The other
     files are fetched as bytes and checked against their SHA-256, so their
     type does not matter;
   - a real 404 for a missing file, never the app's index page.
5. After a deploy, check that the pointer names a catalogue that is served,
   on each host:
   ```bash
   B=https://aac.aurasense.ai/app/voices
   curl -fsS "$B/$(curl -fsS "$B/index.json" | python3 -c 'import json,sys; print(json.load(sys.stdin)["current"])')" | head -c 80
   ```
   The site builds give `index.json` and `LICENSES.md` the build's own
   modification time, because the builder stamps every file with one fixed
   time and rsync compares size and time: otherwise a new pointer of the same
   length would not be copied.
6. The app vendors the same runtime separately, through
   `third_party/moonshine-wasm/vendor.sh` into `/app/vendor/`. This build's
   `runtime/` directory is the copy the pack format describes, and its
   licence entry is the one the Licences screen shows. Serve both, or drop one
   once `neural_voice.rs` settles on a single location.

## Licences

`LICENSES.md` lists every runtime, component and pack, with its licence,
attribution, sources and size. Each `licences/*.txt` file holds the upstream
notices verbatim, fetched at pinned commits, with the full SPDX texts from
`spdx/license-list-data` v3.29.0. Nothing is NC or GPL.

Obligations the app has to meet:

- **Label every neural voice "synthetic voice".** This covers the StyleTTS 2
  pre-trained model condition that Kokoro inherits.
- **Show every attribution on a Licences / Sources screen.** EDRDG, whose
  EDICT data the ja G2P uses, requires the acknowledgement on a separate
  About or Sources screen, not only a launch page.
- **Offer the MPL-2.0 source.** Eigen is linked into `moonshine.wasm` under
  MPL-2.0, whose source is https://gitlab.com/libeigen/eigen.
- **Keep ShareAlike files under their own licence.** These files are
  CC BY-SA:
  - the de and hi dictionaries;
  - the it WikiPron data;
  - the ja EDICT data and the ja RoBERTa model.

  If any of them is modified, the modified version stays CC BY-SA.

Licence evidence was read on 2026-09-29. The URL for each item is recorded
in the catalogue (`licence_url`, `sources`) and in its licence text.