open-standards/aac/aac-rs/tools/ai/README.md

The sentence helper's model

The AAC Reference App (aac-web) can download an on-device language model, on request, and use it to suggest whole sentences from the words a person chose, or replies to what was said to them (crates/aac-web/src/ai_helper.rs). It runs in the browser on the graphics chip (WebGPU), in a worker, with Google's LiteRT-LM web runtime. Nothing typed is sent anywhere, and it works offline once downloaded.

Piece	Where	Pinned by
Models: Gemma 4 E2B and E4B, Google AI Edge's web builds (gemma-4-E2B-it-web.litertlm, 2,008,432,640 bytes; gemma-4-E4B-it-web.litertlm, 2,969,059,328 bytes; Apache-2.0)	/app/ai/ (this tool's output)	Hugging Face revision, size and SHA-256 in build_ai.py MODELS
Runtime: @litert-lm/core 0.17.1 (four WebAssembly builds) and the app's worker	/app/vendor/litert-lm-<version>-<rev>/	npm integrity in third_party/litert-lm/vendor.sh; each file's size and SHA-256 in ai_helper.rs RUNTIME_FILES
Languages, their status, and the examples the model is shown	languages.json → the catalogue	this repository

Build and publish

python3 tools/ai/build_ai.py                    # download the pinned models once (into .cache/), split, write the catalogue
python3 tools/ai/build_ai.py --source ID=FILE   # or use a local copy of a model (its size and SHA-256 are checked)
python3 tools/ai/check_packs.py ../../../aac-ai-packs

The output goes beside the open-standards checkout, in aac-ai-packs/ (outside git; $AAC_AI_DIR or --out override). It holds:

• index.json: a pointer to the current catalogue, served no-store.
• index.<sha12>.json: the catalogue (schema aac-ai-index/1). It lists the models' parts by size and SHA-256, and each tested language's status, model, note and examples. Its committed copy is data/ai/index.json.
• <model-id>/part-NN.<sha12>.bin: each model file in 64 MiB parts. Together the parts are the file, unchanged. Each part is small enough for a tablet's model store to hash in memory, and for Cloudflare to cache.
• LICENSES.md and the licence text.

The site builds (prerender-static.sh here, and aac-aurasense-web's scripts/build-app.sh) vendor the runtime and copy the parts to /app/ai/, after check_packs.py has matched them to the committed catalogue. Set AAC_AI=none to build without them (CI does); a deploy of such a build removes the model from the site.

In the browser

• Download. Settings → Sentence helper. The person sees the size and whether they are on Wi-Fi before anything starts.
• Storage. The worker and the one WebAssembly build this browser runs are stored in the model store (Cache API aac-models-v1), with every part of the model. Each file is SHA-256 checked before it is kept.
• Loading. The worker streams the parts, one after another, from the model store into GPU memory (LiteRT-LM's GPU Artisan backend). The model never sits in JavaScript memory whole.
• When it runs. Only where WebGPU has a hardware adapter that is not refused (ai_helper::fit). Never on the processor.
• Checking it. After download it is loaded once and asked one question. The first load compiles the graphics programs, once. A device that cannot run it says so then, rather than at the person's first request.
• Which model. Each language is served by the smallest model that passed its test. A device keeps one model: the larger one also serves every language the smaller one does, so changing to a language that needs it offers the larger model in place of the smaller (Settings says so, with its size), and never the other way round. The larger model asks for a device with at least 8 GB of memory, the smaller one 4 GB (min_memory_gb).
• Memory. About 1.8 GB of graphics memory for E2B and 3.0 GB for E4B while it is in use. It is let go after 10 minutes without a request, and loaded again (seconds) on the next one.

Measured on 2026-09-30 and 2026-10-01:

	MacBook Pro M5 Max, Chrome 154	iPad Air 11-inch (M3), Safari 27.0.1
Runtime build	litertlm_wasm_internal (relaxed SIMD + JSPI)	litertlm_wasm_compat_internal (JSPI)
Download and check, 2.0 GB	8 s from a local server	101 s over Tailscale
First start, once	about 4 s	about 80 s
Later starts	2.7 s	5.9 s
One answer (3 sentences or 4 replies)	0.4–0.7 s	0.8–1.4 s

Languages

A language is offered only after testing. The test is 20 everyday exchanges (eval/items.json), and each answer is read for whether one suggestion says what the person meant, in good grammar, and how many of the replies are natural. A language is ready on a model when at least 15 of 20 sentences are usable and the readers count at least 60 of 80 replies good on average; it is offered with the smallest model it is ready on. languages.json records the status, model and a plain note for each tested language.

English, Spanish, German and Hindi were read by hand (eval/results-2026-10-01.json). The other 41 languages were read by two independent model readers and, where their verdicts could change the outcome, by a third (Gemma 4 31B, eval/judge.py, which agreed with the hand reading on 75 of 80 verdicts); a sentence is usable when most of its readers say so (eval/results-2026-10-01-languages.json). No native speaker has read these yet.

• Ready with E2B (17): Arabic, Danish, English, French, German, Hindi, Indonesian, Korean, Malay, Persian, Portuguese, Russian, Spanish, Swedish, Turkish, Urdu, Vietnamese.
• Ready with E4B (14): Chinese, Dutch, Gujarati, Hebrew, Italian, Japanese, Marathi, Norwegian, Polish, Punjabi, Tagalog, Telugu, Thai, Ukrainian.
• Not yet (14): Amharic, Bengali, Czech, Finnish, Greek, Hausa, Hungarian, Kannada, Malayalam, Romanian, Swahili, Tamil, Yoruba, Zulu. Their notes say what went wrong.

The same few exchanges fail in nearly every language, with both models: "I'm frustrated, nobody understands me" comes out as "I don't understand", and "no, I just had lunch" loses its reason. They count against every language alike.

To add or retest a language:

1. Write its 20 exchanges and its examples (two sentence examples, one with what the partner said and one without, and two reply examples), in that language, as <code>.json, and merge them: python3 eval/merge_languages.py DIR code. It enters the language as not yet tested.
2. Run the exchanges on each model: LANGUAGES=1 node eval/run.mjs SITE OUT.json code.
3. Read the answers, ideally with a native speaker; eval/judge.py is a second reader, not the only one.
4. Set its status, model and note in languages.json, and rebuild.

The examples are all the model is shown in that language, besides the instructions, which say the language's name.

Updating the runtime or the model

• Runtime. Pin the new @litert-lm/core release in vendor.sh, bump LITERT_LM_VERSION and LITERT_LM_VENDOR_REV in ai_helper.rs, then run vendor.sh and copy the sizes and hashes it prints into RUNTIME_FILES and VENDOR_DIGEST. The model's runtime field in the catalogue must name the new runtime. A model made for another runtime is not offered.
• Model. Change its entry in build_ai.py MODELS (revision, size, SHA-256), test the languages again, rebuild and redeploy. People who have the old model see "A newer version is available"; the update downloads only the parts that changed, and removes the old ones.