open-standards/aac/aac-rs/tools/ae/README.md

# First letters' model (abbreviation expansion)

The AAC Reference App (aac-web) can download a small language model, on
request, that turns the first letter of each word the person types (`iwtgh`)
into whole sentences (`I want to go home`), fitting what their partner said
(`crates/aac-web/src/abbrev.rs`). For people for whom every key press costs
effort (ALS, cerebral palsy, eye gaze, a switch), five letters instead of
seventeen. It runs in the browser on the graphics chip (WebGPU), in a worker,
with the listener's onnxruntime-web. Nothing typed is sent anywhere, it never
speaks by itself, and it works offline once downloaded.

| Piece | Where | Pinned by |
|---|---|---|
| Model: Qwen3-0.6B-Base (Apache-2.0), fine-tuned by AuraSense for first letters, INT4 (block 16) for WebGPU | `/app/ae/` (this tool's output) | each file's size and SHA-256 in `build_ae.py` `FILES` |
| What it learned from: Turk AAC (CC BY 4.0), Synthetic-Persona-Chat (CC BY 4.0), conversations written with Gemma 4 (Apache-2.0) | credited in the catalogue and `LICENSES.md`; nothing of them is shipped | `build_ae.py` `CREDITS` |
| Runtime: the listener's `onnxruntime-web` 1.30.0 JSPI build | `/app/vendor/onnxruntime-web-<version>-<rev>/`, stored once for both | `third_party/onnxruntime-web/vendor.sh`, `listen.rs` `RUNTIME_FILES` |
| The app's worker and `@huggingface/tokenizers` 0.2.0 (Apache-2.0) | `/app/vendor/hf-tokenizers-<version>-<rev>/` | npm integrity in `third_party/hf-tokenizers/vendor.sh`; each file's size and SHA-256 in `abbrev.rs` `RUNTIME_FILES` |
| Languages and the evidence for each | `languages.json` → the catalogue | this repository |

## Build and publish

```bash
python3 tools/ae/build_ae.py --source DIR     # DIR: the model library's onnx/ae-en
python3 tools/ae/check_packs.py ../../../aac-ae-packs
```

`--source` defaults to `$AAC_AE_SOURCE`, else `tools/ae/.cache/ae-en`. Every
file there is checked against its pinned size and SHA-256 before anything is
written, so it does not matter where the files come from.

The output goes beside the open-standards checkout, in `aac-ae-packs/`
(outside git; `$AAC_AE_DIR` or `--out` override). It holds:

- `index.json`: a pointer to the current catalogue, served no-store.
- `index.<sha12>.json`: the catalogue (schema `aac-ae-index/1`). It lists the
  model's graph, its weights, its tokenizer, its configuration and its letter
  tables by size and SHA-256, who made them and what it learned from under
  which licences, and each language's status and evidence. Its committed copy
  is `data/ae/index.json`.
- `ae-en-qwen3-0.6b/<file>.<sha12>.<ext>`: 14 files of at most 64 MiB
  (393 MB), each named by its contents. The weights are the export's seven
  files; one (74 MiB) holds a single tensor, the table of word embeddings,
  which the export could not split, so it is stored as two parts the worker
  reads as one.
- `LICENSES.md` and Qwen3's licence text (`licences/`).

The site builds (`prerender-static.sh` here, and `aac-aurasense-web`'s
`scripts/build-app.sh`) vendor the runtime and the worker and copy the files
to `/app/ae/`, after `check_packs.py` has matched them to the committed
catalogue. Set `AAC_AE=none` to build without them (CI does); a deploy of such
a build removes first letters from the site.

## Where the model comes from

It was made and tested in the model library
(`neuronexus_ai/model_library/gemma4-web/abbrev/`, `LOG.md` for every step
and number): Qwen3-0.6B-Base at revision
`da87bfb608c14b7cf20ba1ce41287e8de496c0cd`, with a LoRA of rank 16 on every
linear layer (mlx-lm 0.32.0) trained on 281,414 examples, then merged:
142,128 exchanges (what a partner said, and a reply of 1 to 12 words), each
written twice as first letters the way the paper does it (each word's first
letter, contractions split at the apostrophe, mid-sentence punctuation kept,
lower case), 73% with the partner's line, 30% without punctuation, 30% with
keyboard slips. Replies in any test set were left out of training. The
prompt is `Partner: …\nLetters: i w t g h\nReply:`.

The browser bundle (`onnx/ae-en/`, `export_web.py`) is the onnxruntime-genai
0.17.1 builder's INT4 export (block 16: WebKit's 4-bit kernel is five times
slower at block 32), with the graph edited (`onnx_patch.py`) so every node
runs on WebGPU: the attention mask's int64 arithmetic is fed in directly as
`seqlens_k` and `total_sequence_length`, the beam search reorders the caches
on the graphics chip (`beam_idx`), and the log-sum-exp of the scores comes
back with them. "All nodes placed on [WebGpuExecutionProvider]. Number of
nodes: 371", in Chrome and in WebKit.

The sentences come from a beam search (8 hypotheses, 5 sentences) that keeps
only tokens fitting the letters still to come (`ae_vocab.*`, built from the
tokenizer), so every sentence offered starts each word with the letter typed,
in order. Punctuation typed must be where it was typed; none typed, and it
may go anywhere. With slips forgiven, a letter may also be a QWERTY neighbour
of the one typed, at a cost of 1 nat each (tuned on the development set).
The worker (`third_party/hf-tokenizers/worker.js`) is the prototype's
`ae-engine.mjs`, unchanged in what it computes.

## In the browser

- **Download.** Settings → First letters. The person sees the size (less
  the runtime, when the listener stored it already) and whether they are on
  Wi-Fi before anything starts.
- **Graphics chip only.** Where the listener runs (`abbrev::fit` is
  `listen::fit`): a hardware WebGPU adapter that is not refused, with
  `shader-f16`, in a browser with JSPI. The session is created with the
  processor fallback switched off (`session.disable_cpu_ep_fallback`).
  Without JSPI, Settings says the browser cannot run it yet.
- **Checking it.** After download it is loaded once and finds sentences once;
  a device that cannot run it says so then.
- **Its own GPU device.** The worker asks the adapter for a device of its
  own and gives it to the runtime; letting the model go releases the session
  and destroys the device (all of its graphics memory back), and only then is
  the worker ended. In WebKit (Playwright's, 2026-10-01), ending a worker
  whose device the runtime's JSPI build still holds ended the page with it.
  The whole path (download, first start, sentences, removal) passes in
  Playwright's WebKit as well as in Chrome.
- **Using it.** In text mode, 🔤 First letters beside the message box. While
  it is on, what is typed is read as first letters, and 250 ms after the
  last key up to five sentences take the word suggestions' place, at their
  height, with "They said…" (the sentence helper's line, shared). Choosing one
  puts it in the box; the person can change it and speak it. It stays on:
  when the box is empty again, what is typed is letters again. Keyboard,
  screen reader (a live status line), and switch scanning reach the row.
- **Memory.** About 0.42 GB of graphics memory while loaded (the weights and
  the search's caches). With the sentence helper (1.8 GB) and the listener
  (1.3 GB), no two load at the same time, and only a computer that says it
  has 8 GB or more keeps them all loaded; every other device, including any
  that does not say (Safari, so every iPad), takes turns. It is let go after
  10 minutes unused.

## Languages

A language is offered only when we measured it on this app's own worker, on
replies it never learned from, and the reply meant was among the five
sentences for at least 50 in 100 of at least 200 replies (`MIN_TOP5`,
`MIN_ITEMS` in `build_ae.py` and `abbrev.rs`). `languages.json` records each
number, where it comes from, and the plain note Settings shows. English is the
only language so far: another needs its own conversations to learn from and
its own test.

The test (`eval/`, 2026-10-01): the model library's 1,176 browser test items
(`onnx/ae-en/check_full.json`), each asked through the app's own worker
(`eval/run.mjs`), on onnxruntime-web 1.30.0 WebGPU in Chrome 154 on an Apple
M5 Max, as the app asks: punctuation where typed; each clean set asked with
slips not forgiven and with them forgiven (the app's default, which decides),
the items typed with slips with them forgiven (and, separately, not). A
reply counts when one of the five reads as it does (case, spacing and final
punctuation aside, the model library's measure); "words" sets punctuation
aside too. Numbers in `eval/summary-2026-10-01.json`; every item, with its
sentences, in the model library's `evals/app_worker_*_2026-10-01.json`
(outside git: Turk Dialogues is for testing only, not ours to publish).

| Set | n | first | among 5 | among 5, words | ms (median) |
|---|---|---|---|---|---|
| **Turk Dialogues test, with what the partner said, slips forgiven (the app's default: decides)** | 252 | 31.7% | 51.2% | 54.4% | 163 |
| the same, slips not forgiven | 252 | 32.9% | 52.8% | 56.3% | 161 |
| the same, without what the partner said | 252 | 21.8% | 33.3% | 34.5% | 172 |
| the same typed with slips, slips forgiven | 252 | 23.4% | 37.3% | 39.7% | 172 |
| the same typed with slips, not forgiven | 252 | 17.5% | 27.0% | 29.0% | 174 |
| Held-out conversations, with the partner's line | 200 | 68.0% | 85.5% | 85.5% | 185 |
| the same typed with slips, slips forgiven | 200 | 47.0% | 66.5% | 69.5% | 196 |
| the same typed with slips, not forgiven | 200 | 26.5% | 33.5% | 34.0% | 215 |
| The Gemma 4 baseline's 20 items | 20 | 65% | 85% | 85% | 172 |

Turk Dialogues (Vertanen 2017) is the set the paper (Cai et al., NAACL 2022)
reports on: a fine-tuned 64B LaMDA found 74.4% among 5, T5-large (770M)
55.2%. Ours is the same split without the authors' typo correction. The
model library's browser check (Chrome, the punctuation rule held strict for
every item) found 56.7% there; the app lets punctuation go anywhere when none
is typed (people skip the keys), and an added comma counts as a miss in the
exact measure, which is why "words" is closer to it.

On the 50-item check (`onnx/ae-en/check.json`: 25 Turk, 25 held-out), the
app's worker found 26 first and 36 among five; the model library found 26 and
37 in Chrome, 27 and 37 in WebKit, and 29 and 36 with the bf16 weights.

Why slips are forgiven unless the person turns it off: with it on, replies
typed with slips were found about twice as often (33.5% → 66.5% held out,
27.0% → 37.3% Turk), and replies typed without slips lost 0.5 to 1.6 points.

To measure (on a machine with a graphics chip; from aac-aurasense-web, for
Playwright):

```bash
bash third_party/onnxruntime-web/vendor.sh SITE
bash third_party/hf-tokenizers/vendor.sh SITE
node tools/ae/eval/run.mjs SITE/vendor/onnxruntime-web-* SITE/vendor/hf-tokenizers-* ../../../aac-ae-packs \
  …/abbrev/onnx/ae-en/check_full.json OUT.json SUMMARY.json
```

Then record the numbers and their source in `languages.json`, and rebuild.

Not measured yet: iPad (WebKit with JSPI), Android. The model library timed
WebKit (Playwright, Safari 26.6 engine, macOS) at a median 431 ms per set of
sentences.

## Updating the runtime or the model

- **Runtime.** It is the listener's: see `tools/asr/README.md`. Its new
  revision makes first letters' record an update too (`Installed::runs`).
  The worker or the tokenizer library: bump `TOKENIZERS_VENDOR_REV` (and
  `TOKENIZERS_VERSION`) in `abbrev.rs`, run `vendor.sh` and copy the sizes,
  hashes and digest it prints into `RUNTIME_FILES` and `VENDOR_DIGEST`.
- **Model.** Rebuild the bundle in the model library, pin its sizes and hashes
  in `FILES`, measure again, rebuild and redeploy. People with the old model
  see "A newer version is available"; the update downloads only the files
  that changed, and removes the old ones once the new ones have run.