# The listener's model ("Hearing your partner") The AAC Reference App (aac-web) can download on-device speech recognition, on request, and use it to write down what someone says to the person, in the sentence helper's "They said…" field, so the helper suggests replies without anyone typing (`crates/aac-web/src/listen.rs`). It runs in the browser on the graphics chip (WebGPU), in a worker, with onnxruntime-web. Nothing heard is recorded or sent anywhere, and it works offline once downloaded. | Piece | Where | Pinned by | |---|---|---| | Speech model: NVIDIA Nemotron 3.5 ASR Streaming 0.6B, soniqo's fp16 ONNX export, graphs edited for WebGPU (OpenMDW-1.1) | `/app/asr/` (this tool's output) | each file's size and SHA-256 in `build_asr.py` `FILES` | | Finding speech: Silero VAD v6.2.3, graph edited for WebGPU (MIT) | the same | the same | | Runtime: `onnxruntime-web` 1.30.0 (its JSPI WebGPU build), ONNX Runtime's licence and third-party notices, and the app's worker | `/app/vendor/onnxruntime-web--/` | npm integrity and the notices' SHA-256 in `third_party/onnxruntime-web/vendor.sh`; each file's size and SHA-256 in `listen.rs` `RUNTIME_FILES` | | Languages, their prompt slots and the evidence for each | `languages.json` → the catalogue | this repository | ## Build and publish ```bash python3 tools/asr/build_asr.py --source DIR # DIR: the edited graphs (asr-proto's build/model) python3 tools/asr/check_packs.py ../../../aac-asr-packs ``` `--source` defaults to `$AAC_ASR_SOURCE`, else `tools/asr/.cache/model`. Every file there is checked against its pinned size and SHA-256 before anything is written, so it does not matter where the files come from. The output goes beside the open-standards checkout, in `aac-asr-packs/` (outside git; `$AAC_ASR_DIR` or `--out` override). It holds: - `index.json`: a pointer to the current catalogue, served no-store. - `index..json`: the catalogue (schema `aac-asr-index/1`). It lists the model's three graphs, their weights and its vocabulary by size and SHA-256, who made them and under which licences, and each language's status, prompt slot and evidence. Its committed copy is `data/asr/index.json`. - `nemotron-3.5-asr-streaming-0.6b/..`: 24 files of at most 64 MiB (1.24 GB), each named by its contents: the encoder graph and its 19 weights files, the decoding step and its one, the VAD, and the vocabulary. - `LICENSES.md` and the licence texts (`licences/`). The site builds (`prerender-static.sh` here, and `aac-aurasense-web`'s `scripts/build-app.sh`) vendor the runtime and copy the files to `/app/asr/`, after `check_packs.py` has matched them to the committed catalogue. Set `AAC_ASR=none` to build without them (CI does); a deploy of such a build removes the listener from the site. ## Where the graphs come from The prototype in the model library (`neuronexus_ai/model_library/gemma4-web/asr-proto/`, `RESULTS.md` for every finding) made them from soniqo's export at revision `76daabfd0aaf5ec6ef1e6640eae3b364af6c9970` (of NVIDIA's model at `ea30d66debe3740a08b573244286791d423d6b3e`) and Silero VAD v6.2.3 (`silero_vad_16k_sequence.onnx`, SHA-256 `9ccdacc4…8a85`). Its tools (`edit_encoder.py`, `merge_mel.py`, `build_decoder_joint.py`, `edit_vad.py`, with `fold.py`, `mask_domain.py`, `rewrite_mask.py`, `split_save.py`) live there, beside the graphs. What they change, and why (each checked against the original on the processor): - **Every node on WebGPU.** onnxruntime-web 1.30's WebGPU provider has no kernel for `ConstantOfShape` or `LogSoftmax`, and puts int64, bool and shape work on the processor. So the shape subgraphs are folded (all shapes are static), the attention mask is rebuilt from int32 arithmetic (evaluated over its whole domain first), the joint's `LogSoftmax` is dropped (the greedy choice is the same), and the decoder, joint and ArgMax become one step that scores a window's four frames at once. - **The sound's front end in the graph.** The log-mel spectrogram (NeMo's, checked to 3.3e-5 against a float64 reference on the GPU itself) is merged into the encoder, which takes 5361 raw samples per 320 ms window. - **Small devices.** The LSTMs are unrolled into MatMul and element-wise nodes: onnxruntime's WebGPU LSTM binds 9 storage buffers, and WebGPU guarantees 8. - **A WebGPU bug.** A 1-D `Conv` with an input channel count that is not a multiple of 4 gives wrong values (onnxruntime-web 1.30): Silero's first convolution (129 channels) is padded with zeros to 132. - **Weights unchanged**, split into files of at most 64 MiB. On 125 FLEURS test clips (25 each in English, Spanish, German, Hindi and French), 112 transcripts on WebGPU were character for character the same as the original graphs computed in fp32 on the processor, and none was worse. ## In the browser - **Download.** Settings → Hearing your partner. The person sees the size and whether they are on Wi-Fi before anything starts. - **Storage.** The worker, the runtime build this browser runs and every file of the model are stored in the model store (Cache API `aac-models-v1`), each SHA-256 checked before it is kept. - **Runtime.** onnxruntime-web's JSPI bundle (16.9 MB with its WebAssembly), imported from the model store, and handed its WebAssembly, in a module worker. One thread. It reads the weights one tensor at a time, holding one 64 MiB file at most, never the whole model. The listener is offered only in browsers with JSPI (`WebAssembly.Suspending`: current Chrome and Edge, and Safari on a current iPad): without it, the runtime would have to read all 1.2 GB into memory before it starts, so the asyncify build is not shipped. - **Graphics chip only.** Where WebGPU has a hardware adapter that is not refused and offers `shader-f16`, in a browser with JSPI (`listen::fit`). Every session is created with the processor fallback switched off (`session.disable_cpu_ep_fallback`): a graph that cannot run wholly on the graphics chip does not load. - **Checking it.** After download it is loaded once and every graph runs once on silence: the first load compiles the graphics programs, and a device that cannot run it says so then. - **Listening.** 🎤 beside "They said…" in the sentence helper. The microphone (echo cancellation, noise suppression, automatic gain) is opened only then, and taken at 16 kHz in an AudioWorklet (resampled there if the browser will not). The VAD runs on every 320 ms window; when someone speaks, the two windows before it and what follows go to the speech model, window by window, and the words show as they come. 1 s of silence ends it: the words go in the box, the microphone is let go, and the helper is asked for replies. A sound with no words in it (a cough, a door) does not end it. The person can stop it any time (what was heard is kept). It also stops by itself, and lets the microphone go, when no words have come for 30 s (a sound without words starts that wait again), after 90 s in all (both counted from when listening starts, once the model has loaded, so a permission prompt nobody answers ends it too; Stop while it loads stops the load at once), when the page is hidden (it does not open the microphone on a hidden page) or closed, or when the microphone goes away. While the app speaks (the device's voice or a downloaded one), and for the audio output's delay and 300 ms after, it hears silence: each 20 ms of sound carries its time on the audio clock, so muting follows when a sound was heard, not when it reached the page. No sound is kept once listening ends. - **Memory.** About 1.3 GB of graphics memory while loaded. With the sentence helper (1.8 GB) the two would hold 3.1 GB, so they never load at the same time, and only a computer that says it has 8 GB or more keeps both loaded. Every other device, including any that does not say (Safari, so every iPad), takes turns: the helper lets go when 🎤 is pressed, and the listener lets go before the helper answers. The listener is let go when the sentence helper closes. Measured in the prototype (Chrome 154, MacBook Pro M5 Max): 35 ms per 320 ms window (9 times faster than speech); the final words 0.6–0.8 s after the speaker stops with the prototype's 500 ms rule (now 1 s: about half a second more), plus up to one window; first load 7–13 s. Not measured yet: iPad, Android. ## Languages A language is offered only when we measured it on this app's own runtime at most 15 errors in 100 words (characters, for Japanese and Korean), and `languages.json` records each number, where it comes from, and the plain note Settings shows. NVIDIA's published rate (its model card: FLEURS, 320 ms chunks, the language given) is kept beside ours for reference; it does not decide. The test (`eval/`, 2026-10-01): the first 25 FLEURS test clips of each language (`google/fleurs` at `70bb2e84`, CC BY 4.0; Spanish is es_419, Arabic ar_eg, Portuguese pt_br), each fed as the microphone would feed it, through the app's own worker on WebGPU (Chrome, Apple GPU): the voice activity detector decides when speech starts and ends, the end-of-speech rule ends it, one utterance per clip, with the prompt slot the app uses. `eval/score.py` counts the errors after a plain normalisation (lower case, no punctuation; Hindi and Arabic spelling variants folded; spaces left out for characters), and with `--no-end` the clips the rule cut at a pause (less than 70% of what was heard with no end at all). Every clip's reference and what was heard are in `eval/results-2026-10-01.json` (as shipped, the 1 s rule), `eval/results-2026-10-01-silence-500ms.json` (the prototype's 500 ms rule) and `eval/results-2026-10-01-no-end-rule.json` (no end at all, for comparison). | | ours (1 s rule, shipped) | ours with 500 ms | NVIDIA's | |---|---|---|---| | **Ready (12)** | | | | | Spanish | 4.5 | 7.5 | 4.4 | | Italian | 6.9 | 16.4 | 4.8 | | Arabic | 8.4 | 9.2 | 12.6 | | English | 9.8 | 12.6 | 8.3 | | Korean (characters) | 10.7 | 10.7 | 7.3 | | Portuguese | 11.4 | 25.1 | 5.8 | | French | 11.7 | 12.4 | 9.8 | | German | 12.9 | 25.0 | 8.8 | | Ukrainian | 13.8 | 15.6 | 14.6 | | Vietnamese | 14.1 | 53.6 | 12.3 | | Russian | 14.3 | 14.3 | 9.9 | | Japanese (characters) | 14.6 | 17.8 | 12.2 | | **Not yet (4)** | | | | | Turkish | 16.2 | 17.0 | 12.1 | | Dutch | 16.2 | 21.7 | 12.2 | | Polish | 17.4 | 17.4 | 17.5 | | Hindi | 17.5 | 25.3 | 7.4 | Why 1 s: with the prototype's 500 ms rule, listening ended at the pause between two sentences of a clip (41 of the 400 clips: Vietnamese 16 of 25, Hindi 5, Portuguese 5, Italian 4, German 3), so in the app the partner was cut off and what they said next was not heard; only 6 languages passed. With 1 s the words come about half a second later, and 4 of the 400 clips are still cut, at pauses longer than a second (Hindi 3, Portuguese 1; with no end at all, Hindi is 10.9). The others are not offered: Chinese, Norwegian, Finnish, Czech, Swedish, Romanian, Danish and Hungarian have NVIDIA rates of 18.9 to 30.9 (not measured here); Greek, Hebrew and Thai NVIDIA trained only as a start for more training; the app's other 18 languages are not the model's. The model is told the language through its prompt slot: the one measured for the language (English 0, Spanish 2, Portuguese 12, French 8, …), for every locale of it. The model has others (en-GB, es-US, pt-PT, fr-CA), but FLEURS has no clips to measure them on (one accent per language), so they are not used. To measure (on a machine with a graphics chip; from aac-aurasense-web, for Playwright): ```bash python3 tools/asr/eval/fetch_fleurs.py CLIPS # 25 clips per language bash third_party/onnxruntime-web/vendor.sh SITE # the runtime, as the app ships it node tools/asr/eval/run.mjs SITE/vendor/onnxruntime-web-* ../../../aac-asr-packs CLIPS OUT.json en es … python3 tools/asr/eval/score.py OUT.json SUMMARY.json ``` Then record the numbers and their source in `languages.json`, and rebuild. ## Updating the runtime or the model - **Runtime.** Pin the new `onnxruntime-web` release in `vendor.sh`, bump `ORT_WEB_VERSION` and `ORT_WEB_VENDOR_REV` in `listen.rs`, run `vendor.sh` and copy the sizes and hashes it prints into `RUNTIME_FILES` and `VENDOR_DIGEST`, and check every graph still runs on the GPU alone. The catalogue's `runtime` must name the new runtime: a model made for another is not offered. - **Model.** Rebuild the graphs, pin their sizes and hashes in `FILES`, test the languages again, rebuild and redeploy. People with the old model see "A newer version is available"; the update downloads only the files that changed, and removes the old ones once the new ones have run.