# The sentence helper's model Express (aac-web) can download an on-device language model, on request, and use it to suggest whole sentences from the words a person chose, or replies to what was said to them (`crates/aac-web/src/ai_helper.rs`). It runs in the browser on the graphics chip (WebGPU), in a worker, with Google's LiteRT-LM web runtime. Nothing typed is sent anywhere, and it works offline once downloaded. | Piece | Where | Pinned by | |---|---|---| | Models: Gemma 4 E2B and E4B, Google AI Edge's web builds (`gemma-4-E2B-it-web.litertlm`, 2,008,432,640 bytes; `gemma-4-E4B-it-web.litertlm`, 2,969,059,328 bytes; Apache-2.0) | `/app/ai/` (this tool's output) | Hugging Face revision, size and SHA-256 in `build_ai.py` `MODELS` | | Runtime: `@litert-lm/core` 0.17.1 (four WebAssembly builds) and the app's worker | `/app/vendor/litert-lm--/` | npm integrity in `third_party/litert-lm/vendor.sh`; each file's size and SHA-256 in `ai_helper.rs` `RUNTIME_FILES` | | Languages, their status, and the examples the model is shown | `languages.json` → the catalogue | this repository | ## Build and publish ```bash python3 tools/ai/build_ai.py # download the pinned models once (into .cache/), split, write the catalogue python3 tools/ai/build_ai.py --source ID=FILE # or use a local copy of a model (its size and SHA-256 are checked) python3 tools/ai/check_packs.py ../../../aac-ai-packs ``` The output goes beside the open-standards checkout, in `aac-ai-packs/` (outside git; `$AAC_AI_DIR` or `--out` override). It holds: - `index.json`: a pointer to the current catalogue, served no-store. - `index..json`: the catalogue (schema `aac-ai-index/1`). It lists the models' parts by size and SHA-256, and each tested language's status, model, note and examples. Its committed copy is `data/ai/index.json`. - `/part-NN..bin`: each model file in 64 MiB parts. Together the parts are the file, unchanged. Each part is small enough for a tablet's model store to hash in memory, and for Cloudflare to cache. - `LICENSES.md` and the licence text. The site builds (`prerender-static.sh` here, and `aac-aurasense-web`'s `scripts/build-app.sh`) vendor the runtime and copy the parts to `/app/ai/`, after `check_packs.py` has matched them to the committed catalogue. Set `AAC_AI=none` to build without them (CI does); a deploy of such a build removes the model from the site. ## In the browser - **Download.** Settings → Sentence helper. The person sees the size and whether they are on Wi-Fi before anything starts. - **Storage.** The worker and the one WebAssembly build this browser runs are stored in the model store (Cache API `aac-models-v1`), with every part of the model. Each file is SHA-256 checked before it is kept. - **Loading.** The worker streams the parts, one after another, from the model store into GPU memory (LiteRT-LM's GPU Artisan backend). The model never sits in JavaScript memory whole. - **When it runs.** Only where WebGPU has a hardware adapter that is not refused (`ai_helper::fit`). Never on the processor. - **Checking it.** After download it is loaded once and asked one question. The first load compiles the graphics programs, once. A device that cannot run it says so then, rather than at the person's first request. - **Which model.** Each language is served by the smallest model that passed its test. A device keeps one model: the larger one also serves every language the smaller one does, so changing to a language that needs it offers the larger model in place of the smaller (Settings says so, with its size), and never the other way round. The larger model asks for a device with at least 8 GB of memory, the smaller one 4 GB (`min_memory_gb`). - **Memory.** About 1.8 GB of graphics memory for E2B and 3.0 GB for E4B while it is in use. It is let go after 10 minutes without a request, and loaded again (seconds) on the next one. Measured on 2026-09-30 and 2026-10-01: | | MacBook Pro M5 Max, Chrome 154 | iPad Air 11-inch (M3), Safari 27.0.1 | |---|---|---| | Runtime build | `litertlm_wasm_internal` (relaxed SIMD + JSPI) | `litertlm_wasm_compat_internal` (JSPI) | | Download and check, 2.0 GB | 8 s from a local server | 101 s over Tailscale | | First start, once | about 4 s | about 80 s | | Later starts | 2.7 s | 5.9 s | | One answer (3 sentences or 4 replies) | 0.4–0.7 s | 0.8–1.4 s | ## Languages A language is offered only after testing. The test is 20 everyday exchanges (`eval/items.json`), and each answer is read for whether one suggestion says what the person meant, in good grammar, and how many of the replies are natural. A language is ready on a model when at least 15 of 20 sentences are usable and the readers count at least 60 of 80 replies good on average; it is offered with the smallest model it is ready on. `languages.json` records the status, model and a plain note for each tested language. English, Spanish, German and Hindi were read by hand (`eval/results-2026-10-01.json`). The other 41 languages were read by two independent model readers and, where their verdicts could change the outcome, by a third (Gemma 4 31B, `eval/judge.py`, which agreed with the hand reading on 75 of 80 verdicts); a sentence is usable when most of its readers say so (`eval/results-2026-10-01-languages.json`). No native speaker has read these yet. - **Ready with E2B (17):** Arabic, Danish, English, French, German, Hindi, Indonesian, Korean, Malay, Persian, Portuguese, Russian, Spanish, Swedish, Turkish, Urdu, Vietnamese. - **Ready with E4B (14):** Chinese, Dutch, Gujarati, Hebrew, Italian, Japanese, Marathi, Norwegian, Polish, Punjabi, Tagalog, Telugu, Thai, Ukrainian. - **Not yet (14):** Amharic, Bengali, Czech, Finnish, Greek, Hausa, Hungarian, Kannada, Malayalam, Romanian, Swahili, Tamil, Yoruba, Zulu. Their notes say what went wrong. The same few exchanges fail in nearly every language, with both models: "I'm frustrated, nobody understands me" comes out as "I don't understand", and "no, I just had lunch" loses its reason. They count against every language alike. To add or retest a language: 1. Write its 20 exchanges and its examples (two sentence examples, one with what the partner said and one without, and two reply examples), in that language, as `.json`, and merge them: `python3 eval/merge_languages.py DIR code`. It enters the language as not yet tested. 2. Run the exchanges on each model: `LANGUAGES=1 node eval/run.mjs SITE OUT.json code`. 3. Read the answers, ideally with a native speaker; `eval/judge.py` is a second reader, not the only one. 4. Set its status, model and note in `languages.json`, and rebuild. The examples are all the model is shown in that language, besides the instructions, which say the language's name. ## Updating the runtime or the model - **Runtime.** Pin the new `@litert-lm/core` release in `vendor.sh`, bump `LITERT_LM_VERSION` and `LITERT_LM_VENDOR_REV` in `ai_helper.rs`, then run `vendor.sh` and copy the sizes and hashes it prints into `RUNTIME_FILES` and `VENDOR_DIGEST`. The model's `runtime` field in the catalogue must name the new runtime. A model made for another runtime is not offered. - **Model.** Change its entry in `build_ai.py` `MODELS` (revision, size, SHA-256), test the languages again, rebuild and redeploy. People who have the old model see "A newer version is available"; the update downloads only the parts that changed, and removes the old ones.