# First letters' model (abbreviation expansion) Express (aac-web) can download a small language model, on request, that turns the first letter of each word the person types (`iwtgh`) into whole sentences (`I want to go home`), fitting what their partner said (`crates/aac-web/src/abbrev.rs`). For people for whom every key press costs effort (ALS, cerebral palsy, eye gaze, a switch), five letters instead of seventeen. It runs in the browser on the graphics chip (WebGPU), in a worker, with the listener's onnxruntime-web. Nothing typed is sent anywhere, it never speaks by itself, and it works offline once downloaded. | Piece | Where | Pinned by | |---|---|---| | Model: Qwen3-0.6B-Base (Apache-2.0), fine-tuned by AuraSense for first letters, INT4 (block 16) for WebGPU | `/app/ae/` (this tool's output) | each file's size and SHA-256 in `build_ae.py` `FILES` | | What it learned from: Turk AAC (CC BY 4.0), Synthetic-Persona-Chat (CC BY 4.0), conversations written with Gemma 4 (Apache-2.0) | credited in the catalogue and `LICENSES.md`; nothing of them is shipped | `build_ae.py` `CREDITS` | | Runtime: the listener's `onnxruntime-web` 1.30.0 JSPI build | `/app/vendor/onnxruntime-web--/`, stored once for both | `third_party/onnxruntime-web/vendor.sh`, `listen.rs` `RUNTIME_FILES` | | The app's worker and `@huggingface/tokenizers` 0.2.0 (Apache-2.0) | `/app/vendor/hf-tokenizers--/` | npm integrity in `third_party/hf-tokenizers/vendor.sh`; each file's size and SHA-256 in `abbrev.rs` `RUNTIME_FILES` | | Languages and the evidence for each | `languages.json` → the catalogue | this repository | ## Build and publish ```bash python3 tools/ae/build_ae.py --source DIR # DIR: the model library's onnx/ae-en python3 tools/ae/check_packs.py ../../../aac-ae-packs ``` `--source` defaults to `$AAC_AE_SOURCE`, else `tools/ae/.cache/ae-en`. Every file there is checked against its pinned size and SHA-256 before anything is written, so it does not matter where the files come from. The output goes beside the open-standards checkout, in `aac-ae-packs/` (outside git; `$AAC_AE_DIR` or `--out` override). It holds: - `index.json`: a pointer to the current catalogue, served no-store. - `index..json`: the catalogue (schema `aac-ae-index/1`). It lists the model's graph, its weights, its tokenizer, its configuration and its letter tables by size and SHA-256, who made them and what it learned from under which licences, and each language's status and evidence. Its committed copy is `data/ae/index.json`. - `ae-en-qwen3-0.6b/..`: 14 files of at most 64 MiB (393 MB), each named by its contents. The weights are the export's seven files; one (74 MiB) holds a single tensor, the table of word embeddings, which the export could not split, so it is stored as two parts the worker reads as one. - `LICENSES.md` and Qwen3's licence text (`licences/`). The site builds (`prerender-static.sh` here, and `aac-aurasense-web`'s `scripts/build-app.sh`) vendor the runtime and the worker and copy the files to `/app/ae/`, after `check_packs.py` has matched them to the committed catalogue. Set `AAC_AE=none` to build without them (CI does); a deploy of such a build removes first letters from the site. ## Where the model comes from It was made and tested in the model library (`neuronexus_ai/model_library/gemma4-web/abbrev/`, `LOG.md` for every step and number): Qwen3-0.6B-Base at revision `da87bfb608c14b7cf20ba1ce41287e8de496c0cd`, with a LoRA of rank 16 on every linear layer (mlx-lm 0.32.0) trained on 281,414 examples, then merged: 142,128 exchanges (what a partner said, and a reply of 1 to 12 words), each written twice as first letters the way the paper does it (each word's first letter, contractions split at the apostrophe, mid-sentence punctuation kept, lower case), 73% with the partner's line, 30% without punctuation, 30% with keyboard slips. Replies in any test set were left out of training. The prompt is `Partner: …\nLetters: i w t g h\nReply:`. The browser bundle (`onnx/ae-en/`, `export_web.py`) is the onnxruntime-genai 0.17.1 builder's INT4 export (block 16: WebKit's 4-bit kernel is five times slower at block 32), with the graph edited (`onnx_patch.py`) so every node runs on WebGPU: the attention mask's int64 arithmetic is fed in directly as `seqlens_k` and `total_sequence_length`, the beam search reorders the caches on the graphics chip (`beam_idx`), and the log-sum-exp of the scores comes back with them. "All nodes placed on [WebGpuExecutionProvider]. Number of nodes: 371", in Chrome and in WebKit. The sentences come from a beam search (8 hypotheses, 5 sentences) that keeps only tokens fitting the letters still to come (`ae_vocab.*`, built from the tokenizer), so every sentence offered starts each word with the letter typed, in order. Punctuation typed must be where it was typed; none typed, and it may go anywhere. With slips forgiven, a letter may also be a QWERTY neighbour of the one typed, at a cost of 1 nat each (tuned on the development set). The worker (`third_party/hf-tokenizers/worker.js`) is the prototype's `ae-engine.mjs`, unchanged in what it computes. ## In the browser - **Download.** Settings → First letters. The person sees the size (less the runtime, when the listener stored it already) and whether they are on Wi-Fi before anything starts. - **Graphics chip only.** Where the listener runs (`abbrev::fit` is `listen::fit`): a hardware WebGPU adapter that is not refused, with `shader-f16`, in a browser with JSPI. The session is created with the processor fallback switched off (`session.disable_cpu_ep_fallback`). Without JSPI, Settings says the browser cannot run it yet. - **Checking it.** After download it is loaded once and finds sentences once; a device that cannot run it says so then. - **Its own GPU device.** The worker asks the adapter for a device of its own and gives it to the runtime; letting the model go releases the session and destroys the device (all of its graphics memory back), and only then is the worker ended. In WebKit (Playwright's, 2026-10-01), ending a worker whose device the runtime's JSPI build still holds ended the page with it. The whole path (download, first start, sentences, removal) passes in Playwright's WebKit as well as in Chrome. - **Using it.** In text mode, 🔤 First letters beside the message box. While it is on, what is typed is read as first letters, and 250 ms after the last key up to five sentences take the word suggestions' place, at their height, with "They said…" (the sentence helper's line, shared). Choosing one puts it in the box; the person can change it and speak it. It stays on: when the box is empty again, what is typed is letters again. Keyboard, screen reader (a live status line), and switch scanning reach the row. - **Memory.** About 0.42 GB of graphics memory while loaded (the weights and the search's caches). With the sentence helper (1.8 GB) and the listener (1.3 GB), no two load at the same time, and only a computer that says it has 8 GB or more keeps them all loaded; every other device, including any that does not say (Safari, so every iPad), takes turns. It is let go after 10 minutes unused. ## Languages A language is offered only when we measured it on this app's own worker, on replies it never learned from, and the reply meant was among the five sentences for at least 50 in 100 of at least 200 replies (`MIN_TOP5`, `MIN_ITEMS` in `build_ae.py` and `abbrev.rs`). `languages.json` records each number, where it comes from, and the plain note Settings shows. English is the only language so far: another needs its own conversations to learn from and its own test. The test (`eval/`, 2026-10-01): the model library's 1,176 browser test items (`onnx/ae-en/check_full.json`), each asked through the app's own worker (`eval/run.mjs`), on onnxruntime-web 1.30.0 WebGPU in Chrome 154 on an Apple M5 Max, as the app asks: punctuation where typed; each clean set asked with slips not forgiven and with them forgiven (the app's default, which decides), the items typed with slips with them forgiven (and, separately, not). A reply counts when one of the five reads as it does (case, spacing and final punctuation aside, the model library's measure); "words" sets punctuation aside too. Numbers in `eval/summary-2026-10-01.json`; every item, with its sentences, in the model library's `evals/app_worker_*_2026-10-01.json` (outside git: Turk Dialogues is for testing only, not ours to publish). | Set | n | first | among 5 | among 5, words | ms (median) | |---|---|---|---|---|---| | **Turk Dialogues test, with what the partner said, slips forgiven (the app's default: decides)** | 252 | 31.7% | 51.2% | 54.4% | 163 | | the same, slips not forgiven | 252 | 32.9% | 52.8% | 56.3% | 161 | | the same, without what the partner said | 252 | 21.8% | 33.3% | 34.5% | 172 | | the same typed with slips, slips forgiven | 252 | 23.4% | 37.3% | 39.7% | 172 | | the same typed with slips, not forgiven | 252 | 17.5% | 27.0% | 29.0% | 174 | | Held-out conversations, with the partner's line | 200 | 68.0% | 85.5% | 85.5% | 185 | | the same typed with slips, slips forgiven | 200 | 47.0% | 66.5% | 69.5% | 196 | | the same typed with slips, not forgiven | 200 | 26.5% | 33.5% | 34.0% | 215 | | The Gemma 4 baseline's 20 items | 20 | 65% | 85% | 85% | 172 | Turk Dialogues (Vertanen 2017) is the set the paper (Cai et al., NAACL 2022) reports on: a fine-tuned 64B LaMDA found 74.4% among 5, T5-large (770M) 55.2%. Ours is the same split without the authors' typo correction. The model library's browser check (Chrome, the punctuation rule held strict for every item) found 56.7% there; the app lets punctuation go anywhere when none is typed (people skip the keys), and an added comma counts as a miss in the exact measure, which is why "words" is closer to it. On the 50-item check (`onnx/ae-en/check.json`: 25 Turk, 25 held-out), the app's worker found 26 first and 36 among five; the model library found 26 and 37 in Chrome, 27 and 37 in WebKit, and 29 and 36 with the bf16 weights. Why slips are forgiven unless the person turns it off: with it on, replies typed with slips were found about twice as often (33.5% → 66.5% held out, 27.0% → 37.3% Turk), and replies typed without slips lost 0.5 to 1.6 points. To measure (on a machine with a graphics chip; from aac-aurasense-web, for Playwright): ```bash bash third_party/onnxruntime-web/vendor.sh SITE bash third_party/hf-tokenizers/vendor.sh SITE node tools/ae/eval/run.mjs SITE/vendor/onnxruntime-web-* SITE/vendor/hf-tokenizers-* ../../../aac-ae-packs \ …/abbrev/onnx/ae-en/check_full.json OUT.json SUMMARY.json ``` Then record the numbers and their source in `languages.json`, and rebuild. Not measured yet: iPad (WebKit with JSPI), Android. The model library timed WebKit (Playwright, Safari 26.6 engine, macOS) at a median 431 ms per set of sentences. ## Updating the runtime or the model - **Runtime.** It is the listener's: see `tools/asr/README.md`. Its new revision makes first letters' record an update too (`Installed::runs`). The worker or the tokenizer library: bump `TOKENIZERS_VENDOR_REV` (and `TOKENIZERS_VERSION`) in `abbrev.rs`, run `vendor.sh` and copy the sizes, hashes and digest it prints into `RUNTIME_FILES` and `VENDOR_DIGEST`. - **Model.** Rebuild the bundle in the model library, pin its sizes and hashes in `FILES`, measure again, rebuild and redeploy. People with the old model see "A newer version is available"; the update downloads only the files that changed, and removes the old ones once the new ones have run.