open-standards/aac/aac-rs/tools/asr

The listener's model ("Hearing your partner")

The AAC Reference App (aac-web) can download on-device speech recognition, on request, and use it to write down what someone says to the person, in the sentence helper's "They said…" field, so the helper suggests replies without anyone typing (crates/aac-web/src/listen.rs). It runs in the browser on the graphics chip (WebGPU), in a worker, with onnxruntime-web. Nothing heard is recorded or sent anywhere, and it works offline once downloaded.

Piece	Where	Pinned by
Speech model: NVIDIA Nemotron 3.5 ASR Streaming 0.6B, soniqo's fp16 ONNX export, graphs edited for WebGPU (OpenMDW-1.1)	/app/asr/ (this tool's output)	each file's size and SHA-256 in build_asr.py FILES
Finding speech: Silero VAD v6.2.3, graph edited for WebGPU (MIT)	the same	the same
Runtime: onnxruntime-web 1.30.0 (its JSPI WebGPU build), ONNX Runtime's licence and third-party notices, and the app's worker	/app/vendor/onnxruntime-web-<version>-<rev>/	npm integrity and the notices' SHA-256 in third_party/onnxruntime-web/vendor.sh; each file's size and SHA-256 in listen.rs RUNTIME_FILES
Languages, their prompt slots and the evidence for each	languages.json → the catalogue	this repository

Build and publish

python3 tools/asr/build_asr.py --source DIR     # DIR: the edited graphs (asr-proto's build/model)
python3 tools/asr/check_packs.py ../../../aac-asr-packs

--source defaults to $AAC_ASR_SOURCE, else tools/asr/.cache/model. Every file there is checked against its pinned size and SHA-256 before anything is written, so it does not matter where the files come from.

The output goes beside the open-standards checkout, in aac-asr-packs/ (outside git; $AAC_ASR_DIR or --out override). It holds:

• index.json: a pointer to the current catalogue, served no-store.
• index.<sha12>.json: the catalogue (schema aac-asr-index/1). It lists the model's three graphs, their weights and its vocabulary by size and SHA-256, who made them and under which licences, and each language's status, prompt slot and evidence. Its committed copy is data/asr/index.json.
• nemotron-3.5-asr-streaming-0.6b/<file>.<sha12>.<ext>: 24 files of at most 64 MiB (1.24 GB), each named by its contents: the encoder graph and its 19 weights files, the decoding step and its one, the VAD, and the vocabulary.
• LICENSES.md and the licence texts (licences/).

The site builds (prerender-static.sh here, and aac-aurasense-web's scripts/build-app.sh) vendor the runtime and copy the files to /app/asr/, after check_packs.py has matched them to the committed catalogue. Set AAC_ASR=none to build without them (CI does); a deploy of such a build removes the listener from the site.

Where the graphs come from

The prototype in the model library (neuronexus_ai/model_library/gemma4-web/asr-proto/, RESULTS.md for every finding) made them from soniqo's export at revision 76daabfd0aaf5ec6ef1e6640eae3b364af6c9970 (of NVIDIA's model at ea30d66debe3740a08b573244286791d423d6b3e) and Silero VAD v6.2.3 (silero_vad_16k_sequence.onnx, SHA-256 9ccdacc4…8a85). Its tools (edit_encoder.py, merge_mel.py, build_decoder_joint.py, edit_vad.py, with fold.py, mask_domain.py, rewrite_mask.py, split_save.py) live there, beside the graphs. What they change, and why (each checked against the original on the processor):

• Every node on WebGPU. onnxruntime-web 1.30's WebGPU provider has no kernel for ConstantOfShape or LogSoftmax, and puts int64, bool and shape work on the processor. So the shape subgraphs are folded (all shapes are static), the attention mask is rebuilt from int32 arithmetic (evaluated over its whole domain first), the joint's LogSoftmax is dropped (the greedy choice is the same), and the decoder, joint and ArgMax become one step that scores a window's four frames at once.
• The sound's front end in the graph. The log-mel spectrogram (NeMo's, checked to 3.3e-5 against a float64 reference on the GPU itself) is merged into the encoder, which takes 5361 raw samples per 320 ms window.
• Small devices. The LSTMs are unrolled into MatMul and element-wise nodes: onnxruntime's WebGPU LSTM binds 9 storage buffers, and WebGPU guarantees 8.
• A WebGPU bug. A 1-D Conv with an input channel count that is not a multiple of 4 gives wrong values (onnxruntime-web 1.30): Silero's first convolution (129 channels) is padded with zeros to 132.
• Weights unchanged, split into files of at most 64 MiB.

On 125 FLEURS test clips (25 each in English, Spanish, German, Hindi and French), 112 transcripts on WebGPU were character for character the same as the original graphs computed in fp32 on the processor, and none was worse.

In the browser

• Download. Settings → Hearing your partner. The person sees the size and whether they are on Wi-Fi before anything starts.
• Storage. The worker, the runtime build this browser runs and every file of the model are stored in the model store (Cache API aac-models-v1), each SHA-256 checked before it is kept.
• Runtime. onnxruntime-web's JSPI bundle (16.9 MB with its WebAssembly), imported from the model store, and handed its WebAssembly, in a module worker. One thread. It reads the weights one tensor at a time, holding one 64 MiB file at most, never the whole model. The listener is offered only in browsers with JSPI (WebAssembly.Suspending: current Chrome and Edge, and Safari on a current iPad): without it, the runtime would have to read all 1.2 GB into memory before it starts, so the asyncify build is not shipped.
• Graphics chip only. Where WebGPU has a hardware adapter that is not refused and offers shader-f16, in a browser with JSPI (listen::fit). Every session is created with the processor fallback switched off (session.disable_cpu_ep_fallback): a graph that cannot run wholly on the graphics chip does not load.
• Checking it. After download it is loaded once and every graph runs once on silence: the first load compiles the graphics programs, and a device that cannot run it says so then.
• Listening. 🎤 beside "They said…" in the sentence helper. The microphone (echo cancellation, noise suppression, automatic gain) is opened only then, and taken at 16 kHz in an AudioWorklet (resampled there if the browser will not). The VAD runs on every 320 ms window; when someone speaks, the two windows before it and what follows go to the speech model, window by window, and the words show as they come. 1 s of silence ends it: the words go in the box, the microphone is let go, and the helper is asked for replies. A sound with no words in it (a cough, a door) does not end it. The person can stop it any time (what was heard is kept). It also stops by itself, and lets the microphone go, when no words have come for 30 s (a sound without words starts that wait again), after 90 s in all (both counted from when listening starts, once the model has loaded, so a permission prompt nobody answers ends it too; Stop while it loads stops the load at once), when the page is hidden (it does not open the microphone on a hidden page) or closed, or when the microphone goes away. While the app speaks (the device's voice or a downloaded one), and for the audio output's delay and 300 ms after, it hears silence: each 20 ms of sound carries its time on the audio clock, so muting follows when a sound was heard, not when it reached the page. No sound is kept once listening ends.
• Memory. About 1.3 GB of graphics memory while loaded. With the sentence helper (1.8 GB) the two would hold 3.1 GB, so they never load at the same time, and only a computer that says it has 8 GB or more keeps both loaded. Every other device, including any that does not say (Safari, so every iPad), takes turns: the helper lets go when 🎤 is pressed, and the listener lets go before the helper answers. The listener is let go when the sentence helper closes.

Measured in the prototype (Chrome 154, MacBook Pro M5 Max): 35 ms per 320 ms window (9 times faster than speech); the final words 0.6–0.8 s after the speaker stops with the prototype's 500 ms rule (now 1 s: about half a second more), plus up to one window; first load 7–13 s. Not measured yet: iPad, Android.

Languages

A language is offered only when we measured it on this app's own runtime at most 15 errors in 100 words (characters, for Japanese and Korean), and languages.json records each number, where it comes from, and the plain note Settings shows. NVIDIA's published rate (its model card: FLEURS, 320 ms chunks, the language given) is kept beside ours for reference; it does not decide.

The test (eval/, 2026-10-01): the first 25 FLEURS test clips of each language (google/fleurs at 70bb2e84, CC BY 4.0; Spanish is es_419, Arabic ar_eg, Portuguese pt_br), each fed as the microphone would feed it, through the app's own worker on WebGPU (Chrome, Apple GPU): the voice activity detector decides when speech starts and ends, the end-of-speech rule ends it, one utterance per clip, with the prompt slot the app uses. eval/score.py counts the errors after a plain normalisation (lower case, no punctuation; Hindi and Arabic spelling variants folded; spaces left out for characters), and with --no-end the clips the rule cut at a pause (less than 70% of what was heard with no end at all). Every clip's reference and what was heard are in eval/results-2026-10-01.json (as shipped, the 1 s rule), eval/results-2026-10-01-silence-500ms.json (the prototype's 500 ms rule) and eval/results-2026-10-01-no-end-rule.json (no end at all, for comparison).

	ours (1 s rule, shipped)	ours with 500 ms	NVIDIA's
Ready (12)
Spanish	4.5	7.5	4.4
Italian	6.9	16.4	4.8
Arabic	8.4	9.2	12.6
English	9.8	12.6	8.3
Korean (characters)	10.7	10.7	7.3
Portuguese	11.4	25.1	5.8
French	11.7	12.4	9.8
German	12.9	25.0	8.8
Ukrainian	13.8	15.6	14.6
Vietnamese	14.1	53.6	12.3
Russian	14.3	14.3	9.9
Japanese (characters)	14.6	17.8	12.2
Not yet (4)
Turkish	16.2	17.0	12.1
Dutch	16.2	21.7	12.2
Polish	17.4	17.4	17.5
Hindi	17.5	25.3	7.4

Why 1 s: with the prototype's 500 ms rule, listening ended at the pause between two sentences of a clip (41 of the 400 clips: Vietnamese 16 of 25, Hindi 5, Portuguese 5, Italian 4, German 3), so in the app the partner was cut off and what they said next was not heard; only 6 languages passed. With 1 s the words come about half a second later, and 4 of the 400 clips are still cut, at pauses longer than a second (Hindi 3, Portuguese 1; with no end at all, Hindi is 10.9).

The others are not offered: Chinese, Norwegian, Finnish, Czech, Swedish, Romanian, Danish and Hungarian have NVIDIA rates of 18.9 to 30.9 (not measured here); Greek, Hebrew and Thai NVIDIA trained only as a start for more training; the app's other 18 languages are not the model's.

The model is told the language through its prompt slot: the one measured for the language (English 0, Spanish 2, Portuguese 12, French 8, …), for every locale of it. The model has others (en-GB, es-US, pt-PT, fr-CA), but FLEURS has no clips to measure them on (one accent per language), so they are not used.

To measure (on a machine with a graphics chip; from aac-aurasense-web, for Playwright):

python3 tools/asr/eval/fetch_fleurs.py CLIPS                       # 25 clips per language
bash third_party/onnxruntime-web/vendor.sh SITE                     # the runtime, as the app ships it
node tools/asr/eval/run.mjs SITE/vendor/onnxruntime-web-* ../../../aac-asr-packs CLIPS OUT.json en es …
python3 tools/asr/eval/score.py OUT.json SUMMARY.json

Then record the numbers and their source in languages.json, and rebuild.

Updating the runtime or the model

• Runtime. Pin the new onnxruntime-web release in vendor.sh, bump ORT_WEB_VERSION and ORT_WEB_VENDOR_REV in listen.rs, run vendor.sh and copy the sizes and hashes it prints into RUNTIME_FILES and VENDOR_DIGEST, and check every graph still runs on the GPU alone. The catalogue's runtime must name the new runtime: a model made for another is not offered.
• Model. Rebuild the graphs, pin their sizes and hashes in FILES, test the languages again, rebuild and redeploy. People with the old model see "A newer version is available"; the update downloads only the files that changed, and removes the old ones once the new ones have run.