open-standards/aac/aac-rs/tools/ae/README.md

First letters' model (abbreviation expansion)

The AAC Reference App (aac-web) can download a small language model, on request, that turns the first letter of each word the person types (iwtgh) into whole sentences (I want to go home), fitting what their partner said (crates/aac-web/src/abbrev.rs). For people for whom every key press costs effort (ALS, cerebral palsy, eye gaze, a switch), five letters instead of seventeen. It runs in the browser on the graphics chip (WebGPU), in a worker, with the listener's onnxruntime-web. Nothing typed is sent anywhere, it never speaks by itself, and it works offline once downloaded.

Piece	Where	Pinned by
Model: Qwen3-0.6B-Base (Apache-2.0), fine-tuned by AuraSense for first letters, INT4 (block 16) for WebGPU	/app/ae/ (this tool's output)	each file's size and SHA-256 in build_ae.py FILES
What it learned from: Turk AAC (CC BY 4.0), Synthetic-Persona-Chat (CC BY 4.0), conversations written with Gemma 4 (Apache-2.0)	credited in the catalogue and LICENSES.md; nothing of them is shipped	build_ae.py CREDITS
Runtime: the listener's onnxruntime-web 1.30.0 JSPI build	/app/vendor/onnxruntime-web-<version>-<rev>/, stored once for both	third_party/onnxruntime-web/vendor.sh, listen.rs RUNTIME_FILES
The app's worker and @huggingface/tokenizers 0.2.0 (Apache-2.0)	/app/vendor/hf-tokenizers-<version>-<rev>/	npm integrity in third_party/hf-tokenizers/vendor.sh; each file's size and SHA-256 in abbrev.rs RUNTIME_FILES
Languages and the evidence for each	languages.json → the catalogue	this repository

Build and publish

python3 tools/ae/build_ae.py --source DIR     # DIR: the model library's onnx/ae-en
python3 tools/ae/check_packs.py ../../../aac-ae-packs

--source defaults to $AAC_AE_SOURCE, else tools/ae/.cache/ae-en. Every file there is checked against its pinned size and SHA-256 before anything is written, so it does not matter where the files come from.

The output goes beside the open-standards checkout, in aac-ae-packs/ (outside git; $AAC_AE_DIR or --out override). It holds:

• index.json: a pointer to the current catalogue, served no-store.
• index.<sha12>.json: the catalogue (schema aac-ae-index/1). It lists the model's graph, its weights, its tokenizer, its configuration and its letter tables by size and SHA-256, who made them and what it learned from under which licences, and each language's status and evidence. Its committed copy is data/ae/index.json.
• ae-en-qwen3-0.6b/<file>.<sha12>.<ext>: 14 files of at most 64 MiB (393 MB), each named by its contents. The weights are the export's seven files; one (74 MiB) holds a single tensor, the table of word embeddings, which the export could not split, so it is stored as two parts the worker reads as one.
• LICENSES.md and Qwen3's licence text (licences/).

The site builds (prerender-static.sh here, and aac-aurasense-web's scripts/build-app.sh) vendor the runtime and the worker and copy the files to /app/ae/, after check_packs.py has matched them to the committed catalogue. Set AAC_AE=none to build without them (CI does); a deploy of such a build removes first letters from the site.

Where the model comes from

It was made and tested in the model library (neuronexus_ai/model_library/gemma4-web/abbrev/, LOG.md for every step and number): Qwen3-0.6B-Base at revision da87bfb608c14b7cf20ba1ce41287e8de496c0cd, with a LoRA of rank 16 on every linear layer (mlx-lm 0.32.0) trained on 281,414 examples, then merged: 142,128 exchanges (what a partner said, and a reply of 1 to 12 words), each written twice as first letters the way the paper does it (each word's first letter, contractions split at the apostrophe, mid-sentence punctuation kept, lower case), 73% with the partner's line, 30% without punctuation, 30% with keyboard slips. Replies in any test set were left out of training. The prompt is Partner: …\nLetters: i w t g h\nReply:.

The browser bundle (onnx/ae-en/, export_web.py) is the onnxruntime-genai 0.17.1 builder's INT4 export (block 16: WebKit's 4-bit kernel is five times slower at block 32), with the graph edited (onnx_patch.py) so every node runs on WebGPU: the attention mask's int64 arithmetic is fed in directly as seqlens_k and total_sequence_length, the beam search reorders the caches on the graphics chip (beam_idx), and the log-sum-exp of the scores comes back with them. "All nodes placed on [WebGpuExecutionProvider]. Number of nodes: 371", in Chrome and in WebKit.

The sentences come from a beam search (8 hypotheses, 5 sentences) that keeps only tokens fitting the letters still to come (ae_vocab.*, built from the tokenizer), so every sentence offered starts each word with the letter typed, in order. Punctuation typed must be where it was typed; none typed, and it may go anywhere. With slips forgiven, a letter may also be a QWERTY neighbour of the one typed, at a cost of 1 nat each (tuned on the development set). The worker (third_party/hf-tokenizers/worker.js) is the prototype's ae-engine.mjs, unchanged in what it computes.

In the browser

• Download. Settings → First letters. The person sees the size (less the runtime, when the listener stored it already) and whether they are on Wi-Fi before anything starts.
• Graphics chip only. Where the listener runs (abbrev::fit is listen::fit): a hardware WebGPU adapter that is not refused, with shader-f16, in a browser with JSPI. The session is created with the processor fallback switched off (session.disable_cpu_ep_fallback). Without JSPI, Settings says the browser cannot run it yet.
• Checking it. After download it is loaded once and finds sentences once; a device that cannot run it says so then.
• Its own GPU device. The worker asks the adapter for a device of its own and gives it to the runtime; letting the model go releases the session and destroys the device (all of its graphics memory back), and only then is the worker ended. In WebKit (Playwright's, 2026-10-01), ending a worker whose device the runtime's JSPI build still holds ended the page with it. The whole path (download, first start, sentences, removal) passes in Playwright's WebKit as well as in Chrome.
• Using it. In text mode, 🔤 First letters beside the message box. While it is on, what is typed is read as first letters, and 250 ms after the last key up to five sentences take the word suggestions' place, at their height, with "They said…" (the sentence helper's line, shared). Choosing one puts it in the box; the person can change it and speak it. It stays on: when the box is empty again, what is typed is letters again. Keyboard, screen reader (a live status line), and switch scanning reach the row.
• Memory. About 0.42 GB of graphics memory while loaded (the weights and the search's caches). With the sentence helper (1.8 GB) and the listener (1.3 GB), no two load at the same time, and only a computer that says it has 8 GB or more keeps them all loaded; every other device, including any that does not say (Safari, so every iPad), takes turns. It is let go after 10 minutes unused.

Languages

A language is offered only when we measured it on this app's own worker, on replies it never learned from, and the reply meant was among the five sentences for at least 50 in 100 of at least 200 replies (MIN_TOP5, MIN_ITEMS in build_ae.py and abbrev.rs). languages.json records each number, where it comes from, and the plain note Settings shows. English is the only language so far: another needs its own conversations to learn from and its own test.

The test (eval/, 2026-10-01): the model library's 1,176 browser test items (onnx/ae-en/check_full.json), each asked through the app's own worker (eval/run.mjs), on onnxruntime-web 1.30.0 WebGPU in Chrome 154 on an Apple M5 Max, as the app asks: punctuation where typed; each clean set asked with slips not forgiven and with them forgiven (the app's default, which decides), the items typed with slips with them forgiven (and, separately, not). A reply counts when one of the five reads as it does (case, spacing and final punctuation aside, the model library's measure); "words" sets punctuation aside too. Numbers in eval/summary-2026-10-01.json; every item, with its sentences, in the model library's evals/app_worker_*_2026-10-01.json (outside git: Turk Dialogues is for testing only, not ours to publish).

Set	n	first	among 5	among 5, words	ms (median)
Turk Dialogues test, with what the partner said, slips forgiven (the app's default: decides)	252	31.7%	51.2%	54.4%	163
the same, slips not forgiven	252	32.9%	52.8%	56.3%	161
the same, without what the partner said	252	21.8%	33.3%	34.5%	172
the same typed with slips, slips forgiven	252	23.4%	37.3%	39.7%	172
the same typed with slips, not forgiven	252	17.5%	27.0%	29.0%	174
Held-out conversations, with the partner's line	200	68.0%	85.5%	85.5%	185
the same typed with slips, slips forgiven	200	47.0%	66.5%	69.5%	196
the same typed with slips, not forgiven	200	26.5%	33.5%	34.0%	215
The Gemma 4 baseline's 20 items	20	65%	85%	85%	172

Turk Dialogues (Vertanen 2017) is the set the paper (Cai et al., NAACL 2022) reports on: a fine-tuned 64B LaMDA found 74.4% among 5, T5-large (770M) 55.2%. Ours is the same split without the authors' typo correction. The model library's browser check (Chrome, the punctuation rule held strict for every item) found 56.7% there; the app lets punctuation go anywhere when none is typed (people skip the keys), and an added comma counts as a miss in the exact measure, which is why "words" is closer to it.

On the 50-item check (onnx/ae-en/check.json: 25 Turk, 25 held-out), the app's worker found 26 first and 36 among five; the model library found 26 and 37 in Chrome, 27 and 37 in WebKit, and 29 and 36 with the bf16 weights.

Why slips are forgiven unless the person turns it off: with it on, replies typed with slips were found about twice as often (33.5% → 66.5% held out, 27.0% → 37.3% Turk), and replies typed without slips lost 0.5 to 1.6 points.

To measure (on a machine with a graphics chip; from aac-aurasense-web, for Playwright):

bash third_party/onnxruntime-web/vendor.sh SITE
bash third_party/hf-tokenizers/vendor.sh SITE
node tools/ae/eval/run.mjs SITE/vendor/onnxruntime-web-* SITE/vendor/hf-tokenizers-* ../../../aac-ae-packs \
  …/abbrev/onnx/ae-en/check_full.json OUT.json SUMMARY.json

Then record the numbers and their source in languages.json, and rebuild.

Not measured yet: iPad (WebKit with JSPI), Android. The model library timed WebKit (Playwright, Safari 26.6 engine, macOS) at a median 431 ms per set of sentences.

Updating the runtime or the model

• Runtime. It is the listener's: see tools/asr/README.md. Its new revision makes first letters' record an update too (Installed::runs). The worker or the tokenizer library: bump TOKENIZERS_VENDOR_REV (and TOKENIZERS_VERSION) in abbrev.rs, run vendor.sh and copy the sizes, hashes and digest it prints into RUNTIME_FILES and VENDOR_DIGEST.
• Model. Rebuild the bundle in the model library, pin its sizes and hashes in FILES, measure again, rebuild and redeploy. People with the old model see "A newer version is available"; the update downloads only the files that changed, and removes the old ones once the new ones have run.