8306784d39
"Express" is also TD Snap's page-set name in the same field. The standard's reference app is now called the AAC Reference App: READMEs, crate docs, tool docs and the page-set design notes (which now say it was renamed). Its default title, without AAC_APP_TITLE, and its manifest name are "AAC — reference app". Code identifiers are unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]>
229 lines
12 KiB
Markdown
229 lines
12 KiB
Markdown
# The listener's model ("Hearing your partner")
|
||
|
||
The AAC Reference App (aac-web) can download on-device speech recognition, on
|
||
request, and use it to write down what someone says to the person, in the
|
||
sentence helper's "They said…" field, so the helper suggests replies without
|
||
anyone typing (`crates/aac-web/src/listen.rs`). It runs in the browser on the
|
||
graphics chip (WebGPU), in a worker, with onnxruntime-web. Nothing heard is
|
||
recorded or sent anywhere, and it works offline once downloaded.
|
||
|
||
| Piece | Where | Pinned by |
|
||
|---|---|---|
|
||
| Speech model: NVIDIA Nemotron 3.5 ASR Streaming 0.6B, soniqo's fp16 ONNX export, graphs edited for WebGPU (OpenMDW-1.1) | `/app/asr/` (this tool's output) | each file's size and SHA-256 in `build_asr.py` `FILES` |
|
||
| Finding speech: Silero VAD v6.2.3, graph edited for WebGPU (MIT) | the same | the same |
|
||
| Runtime: `onnxruntime-web` 1.30.0 (its JSPI WebGPU build), ONNX Runtime's licence and third-party notices, and the app's worker | `/app/vendor/onnxruntime-web-<version>-<rev>/` | npm integrity and the notices' SHA-256 in `third_party/onnxruntime-web/vendor.sh`; each file's size and SHA-256 in `listen.rs` `RUNTIME_FILES` |
|
||
| Languages, their prompt slots and the evidence for each | `languages.json` → the catalogue | this repository |
|
||
|
||
## Build and publish
|
||
|
||
```bash
|
||
python3 tools/asr/build_asr.py --source DIR # DIR: the edited graphs (asr-proto's build/model)
|
||
python3 tools/asr/check_packs.py ../../../aac-asr-packs
|
||
```
|
||
|
||
`--source` defaults to `$AAC_ASR_SOURCE`, else `tools/asr/.cache/model`.
|
||
Every file there is checked against its pinned size and SHA-256 before
|
||
anything is written, so it does not matter where the files come from.
|
||
|
||
The output goes beside the open-standards checkout, in `aac-asr-packs/`
|
||
(outside git; `$AAC_ASR_DIR` or `--out` override). It holds:
|
||
|
||
- `index.json`: a pointer to the current catalogue, served no-store.
|
||
- `index.<sha12>.json`: the catalogue (schema `aac-asr-index/1`). It lists the
|
||
model's three graphs, their weights and its vocabulary by size and SHA-256,
|
||
who made them and under which licences, and each language's status, prompt
|
||
slot and evidence. Its committed copy is `data/asr/index.json`.
|
||
- `nemotron-3.5-asr-streaming-0.6b/<file>.<sha12>.<ext>`: 24 files of at
|
||
most 64 MiB (1.24 GB), each named by its contents: the encoder graph and
|
||
its 19 weights files, the decoding step and its one, the VAD, and the
|
||
vocabulary.
|
||
- `LICENSES.md` and the licence texts (`licences/`).
|
||
|
||
The site builds (`prerender-static.sh` here, and `aac-aurasense-web`'s
|
||
`scripts/build-app.sh`) vendor the runtime and copy the files to `/app/asr/`,
|
||
after `check_packs.py` has matched them to the committed catalogue. Set
|
||
`AAC_ASR=none` to build without them (CI does); a deploy of such a build
|
||
removes the listener from the site.
|
||
|
||
## Where the graphs come from
|
||
|
||
The prototype in the model library
|
||
(`neuronexus_ai/model_library/gemma4-web/asr-proto/`, `RESULTS.md` for every
|
||
finding) made them from soniqo's export at revision
|
||
`76daabfd0aaf5ec6ef1e6640eae3b364af6c9970` (of NVIDIA's model at
|
||
`ea30d66debe3740a08b573244286791d423d6b3e`) and Silero VAD v6.2.3
|
||
(`silero_vad_16k_sequence.onnx`, SHA-256 `9ccdacc4…8a85`). Its tools
|
||
(`edit_encoder.py`, `merge_mel.py`, `build_decoder_joint.py`, `edit_vad.py`,
|
||
with `fold.py`, `mask_domain.py`, `rewrite_mask.py`, `split_save.py`) live
|
||
there, beside the graphs. What they change, and why (each checked against the
|
||
original on the processor):
|
||
|
||
- **Every node on WebGPU.** onnxruntime-web 1.30's WebGPU provider has no
|
||
kernel for `ConstantOfShape` or `LogSoftmax`, and puts int64, bool and
|
||
shape work on the processor. So the shape subgraphs are folded (all
|
||
shapes are static), the attention mask is rebuilt from int32 arithmetic
|
||
(evaluated over its whole domain first), the joint's `LogSoftmax` is
|
||
dropped (the greedy choice is the same), and the decoder, joint and ArgMax
|
||
become one step that scores a window's four frames at once.
|
||
- **The sound's front end in the graph.** The log-mel spectrogram (NeMo's,
|
||
checked to 3.3e-5 against a float64 reference on the GPU itself) is merged
|
||
into the encoder, which takes 5361 raw samples per 320 ms window.
|
||
- **Small devices.** The LSTMs are unrolled into MatMul and element-wise
|
||
nodes: onnxruntime's WebGPU LSTM binds 9 storage buffers, and WebGPU
|
||
guarantees 8.
|
||
- **A WebGPU bug.** A 1-D `Conv` with an input channel count that is not a
|
||
multiple of 4 gives wrong values (onnxruntime-web 1.30): Silero's first
|
||
convolution (129 channels) is padded with zeros to 132.
|
||
- **Weights unchanged**, split into files of at most 64 MiB.
|
||
|
||
On 125 FLEURS test clips (25 each in English, Spanish, German, Hindi and
|
||
French), 112 transcripts on WebGPU were character for character the same as
|
||
the original graphs computed in fp32 on the processor, and none was worse.
|
||
|
||
## In the browser
|
||
|
||
- **Download.** Settings → Hearing your partner. The person sees the size and
|
||
whether they are on Wi-Fi before anything starts.
|
||
- **Storage.** The worker, the runtime build this browser runs and every file
|
||
of the model are stored in the model store (Cache API `aac-models-v1`),
|
||
each SHA-256 checked before it is kept.
|
||
- **Runtime.** onnxruntime-web's JSPI bundle (16.9 MB with its
|
||
WebAssembly), imported from the model store, and handed its WebAssembly,
|
||
in a module worker. One thread. It reads the weights one tensor at a time,
|
||
holding one 64 MiB file at most, never the whole model. The listener is
|
||
offered only in browsers with JSPI (`WebAssembly.Suspending`: current
|
||
Chrome and Edge, and Safari on a current iPad): without it, the runtime
|
||
would have to read all 1.2 GB into memory before it starts, so the
|
||
asyncify build is not shipped.
|
||
- **Graphics chip only.** Where WebGPU has a hardware adapter that is not
|
||
refused and offers `shader-f16`, in a browser with JSPI (`listen::fit`).
|
||
Every session is created
|
||
with the processor fallback switched off
|
||
(`session.disable_cpu_ep_fallback`): a graph that cannot run wholly on the
|
||
graphics chip does not load.
|
||
- **Checking it.** After download it is loaded once and every graph runs
|
||
once on silence: the first load compiles the graphics programs, and a
|
||
device that cannot run it says so then.
|
||
- **Listening.** 🎤 beside "They said…" in the sentence helper. The
|
||
microphone (echo cancellation, noise suppression, automatic gain) is opened
|
||
only then, and taken at 16 kHz in an AudioWorklet (resampled there if the
|
||
browser will not). The VAD runs on every 320 ms window; when someone
|
||
speaks, the two windows before it and what follows go to the speech model,
|
||
window by window, and the words show as they come. 1 s of silence ends
|
||
it: the words go in the box, the microphone is let go, and the helper is
|
||
asked for replies. A sound with no words in it (a cough, a door) does not
|
||
end it. The person can stop it any time (what was heard is kept). It also
|
||
stops by itself, and lets the microphone go, when no words have come for
|
||
30 s (a sound without words starts that wait again), after 90 s in all
|
||
(both counted from when listening starts, once the model has loaded, so a
|
||
permission prompt nobody answers ends it too; Stop while it loads stops
|
||
the load at once), when the page is hidden (it does not open the microphone on a
|
||
hidden page) or closed, or when the microphone goes away.
|
||
While the app speaks (the device's voice or a downloaded one), and for the
|
||
audio output's delay and 300 ms after, it hears silence: each 20 ms of
|
||
sound carries its time on the audio clock, so muting follows when a sound
|
||
was heard, not when it reached the page. No sound is kept once listening
|
||
ends.
|
||
- **Memory.** About 1.3 GB of graphics memory while loaded. With the
|
||
sentence helper (1.8 GB) the two would hold 3.1 GB, so they never load at
|
||
the same time, and only a computer that says it has 8 GB or more keeps
|
||
both loaded. Every other device, including any that does not say (Safari,
|
||
so every iPad), takes turns: the helper lets go when 🎤 is pressed, and the
|
||
listener lets go before the helper answers. The listener is let go when
|
||
the sentence helper closes.
|
||
|
||
Measured in the prototype (Chrome 154, MacBook Pro M5 Max): 35 ms per 320 ms
|
||
window (9 times faster than speech); the final words 0.6–0.8 s after the
|
||
speaker stops with the prototype's 500 ms rule (now 1 s: about half a second
|
||
more), plus up to one window; first load 7–13 s.
|
||
Not measured yet: iPad, Android.
|
||
|
||
## Languages
|
||
|
||
A language is offered only when we measured it on this app's own runtime
|
||
at most 15 errors in 100 words (characters, for Japanese and Korean), and
|
||
`languages.json` records each number, where it comes from, and the plain
|
||
note Settings shows. NVIDIA's published rate (its model card: FLEURS, 320 ms
|
||
chunks, the language given) is kept beside ours for reference; it does not
|
||
decide.
|
||
|
||
The test (`eval/`, 2026-10-01): the first 25 FLEURS test clips of each
|
||
language (`google/fleurs` at `70bb2e84`, CC BY 4.0; Spanish is es_419,
|
||
Arabic ar_eg, Portuguese pt_br), each fed as the microphone would feed it,
|
||
through the app's own worker on WebGPU (Chrome, Apple GPU): the voice
|
||
activity detector decides when speech starts and ends, the end-of-speech
|
||
rule ends it, one utterance per clip, with the prompt slot the app uses.
|
||
`eval/score.py` counts the errors after a plain normalisation (lower case,
|
||
no punctuation; Hindi and Arabic spelling variants folded; spaces left out
|
||
for characters), and with `--no-end` the clips the rule cut at a pause (less
|
||
than 70% of what was heard with no end at all). Every clip's reference and what was heard are in
|
||
`eval/results-2026-10-01.json` (as shipped, the 1 s rule),
|
||
`eval/results-2026-10-01-silence-500ms.json` (the prototype's 500 ms rule)
|
||
and `eval/results-2026-10-01-no-end-rule.json` (no end at all, for
|
||
comparison).
|
||
|
||
| | ours (1 s rule, shipped) | ours with 500 ms | NVIDIA's |
|
||
|---|---|---|---|
|
||
| **Ready (12)** | | | |
|
||
| Spanish | 4.5 | 7.5 | 4.4 |
|
||
| Italian | 6.9 | 16.4 | 4.8 |
|
||
| Arabic | 8.4 | 9.2 | 12.6 |
|
||
| English | 9.8 | 12.6 | 8.3 |
|
||
| Korean (characters) | 10.7 | 10.7 | 7.3 |
|
||
| Portuguese | 11.4 | 25.1 | 5.8 |
|
||
| French | 11.7 | 12.4 | 9.8 |
|
||
| German | 12.9 | 25.0 | 8.8 |
|
||
| Ukrainian | 13.8 | 15.6 | 14.6 |
|
||
| Vietnamese | 14.1 | 53.6 | 12.3 |
|
||
| Russian | 14.3 | 14.3 | 9.9 |
|
||
| Japanese (characters) | 14.6 | 17.8 | 12.2 |
|
||
| **Not yet (4)** | | | |
|
||
| Turkish | 16.2 | 17.0 | 12.1 |
|
||
| Dutch | 16.2 | 21.7 | 12.2 |
|
||
| Polish | 17.4 | 17.4 | 17.5 |
|
||
| Hindi | 17.5 | 25.3 | 7.4 |
|
||
|
||
Why 1 s: with the prototype's 500 ms rule, listening ended at the pause
|
||
between two sentences of a clip (41 of the 400 clips: Vietnamese 16 of
|
||
25, Hindi 5, Portuguese 5, Italian 4, German 3), so in the app the partner
|
||
was cut off and what
|
||
they said next was not heard; only 6 languages passed. With 1 s the words
|
||
come about half a second later, and 4 of the 400 clips are still cut, at
|
||
pauses longer than a second (Hindi 3, Portuguese 1; with no end at all,
|
||
Hindi is 10.9).
|
||
|
||
The others are not offered: Chinese, Norwegian, Finnish, Czech, Swedish,
|
||
Romanian, Danish and Hungarian have NVIDIA rates of 18.9 to 30.9 (not
|
||
measured here); Greek, Hebrew and Thai NVIDIA trained only as a start for
|
||
more training; the app's other 18 languages are not the model's.
|
||
|
||
The model is told the language through its prompt slot: the one measured
|
||
for the language (English 0, Spanish 2, Portuguese 12, French 8, …), for
|
||
every locale of it. The model has others (en-GB, es-US, pt-PT, fr-CA), but
|
||
FLEURS has no clips to measure them on (one accent per language), so they
|
||
are not used.
|
||
|
||
To measure (on a machine with a graphics chip; from aac-aurasense-web, for
|
||
Playwright):
|
||
|
||
```bash
|
||
python3 tools/asr/eval/fetch_fleurs.py CLIPS # 25 clips per language
|
||
bash third_party/onnxruntime-web/vendor.sh SITE # the runtime, as the app ships it
|
||
node tools/asr/eval/run.mjs SITE/vendor/onnxruntime-web-* ../../../aac-asr-packs CLIPS OUT.json en es …
|
||
python3 tools/asr/eval/score.py OUT.json SUMMARY.json
|
||
```
|
||
|
||
Then record the numbers and their source in `languages.json`, and rebuild.
|
||
|
||
## Updating the runtime or the model
|
||
|
||
- **Runtime.** Pin the new `onnxruntime-web` release in `vendor.sh`, bump
|
||
`ORT_WEB_VERSION` and `ORT_WEB_VENDOR_REV` in `listen.rs`, run `vendor.sh`
|
||
and copy the sizes and hashes it prints into `RUNTIME_FILES` and
|
||
`VENDOR_DIGEST`, and check every graph still runs on the GPU alone. The
|
||
catalogue's `runtime` must name the new runtime: a model made for another is
|
||
not offered.
|
||
- **Model.** Rebuild the graphs, pin their sizes and hashes in `FILES`, test
|
||
the languages again, rebuild and redeploy. People with the old model see
|
||
"A newer version is available"; the update downloads only the files that
|
||
changed, and removes the old ones once the new ones have run.
|