8306784d39
"Express" is also TD Snap's page-set name in the same field. The standard's reference app is now called the AAC Reference App: READMEs, crate docs, tool docs and the page-set design notes (which now say it was renamed). Its default title, without AAC_APP_TITLE, and its manifest name are "AAC — reference app". Code identifiers are unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]>
132 lines
7.4 KiB
Markdown
132 lines
7.4 KiB
Markdown
# The sentence helper's model
|
||
|
||
The AAC Reference App (aac-web) can download an on-device language model, on
|
||
request, and use it to suggest whole sentences from the words a person chose,
|
||
or replies to what was said to them (`crates/aac-web/src/ai_helper.rs`). It
|
||
runs in the browser on the graphics chip (WebGPU), in a worker, with Google's
|
||
LiteRT-LM web runtime. Nothing typed is sent anywhere, and it works offline
|
||
once downloaded.
|
||
|
||
| Piece | Where | Pinned by |
|
||
|---|---|---|
|
||
| Models: Gemma 4 E2B and E4B, Google AI Edge's web builds (`gemma-4-E2B-it-web.litertlm`, 2,008,432,640 bytes; `gemma-4-E4B-it-web.litertlm`, 2,969,059,328 bytes; Apache-2.0) | `/app/ai/` (this tool's output) | Hugging Face revision, size and SHA-256 in `build_ai.py` `MODELS` |
|
||
| Runtime: `@litert-lm/core` 0.17.1 (four WebAssembly builds) and the app's worker | `/app/vendor/litert-lm-<version>-<rev>/` | npm integrity in `third_party/litert-lm/vendor.sh`; each file's size and SHA-256 in `ai_helper.rs` `RUNTIME_FILES` |
|
||
| Languages, their status, and the examples the model is shown | `languages.json` → the catalogue | this repository |
|
||
|
||
## Build and publish
|
||
|
||
```bash
|
||
python3 tools/ai/build_ai.py # download the pinned models once (into .cache/), split, write the catalogue
|
||
python3 tools/ai/build_ai.py --source ID=FILE # or use a local copy of a model (its size and SHA-256 are checked)
|
||
python3 tools/ai/check_packs.py ../../../aac-ai-packs
|
||
```
|
||
|
||
The output goes beside the open-standards checkout, in `aac-ai-packs/` (outside
|
||
git; `$AAC_AI_DIR` or `--out` override). It holds:
|
||
|
||
- `index.json`: a pointer to the current catalogue, served no-store.
|
||
- `index.<sha12>.json`: the catalogue (schema `aac-ai-index/1`). It lists the
|
||
models' parts by size and SHA-256, and each tested language's status, model,
|
||
note and examples. Its committed copy is `data/ai/index.json`.
|
||
- `<model-id>/part-NN.<sha12>.bin`: each model file in 64 MiB parts.
|
||
Together the parts are the file, unchanged. Each part is small enough for a
|
||
tablet's model store to hash in memory, and for Cloudflare to cache.
|
||
- `LICENSES.md` and the licence text.
|
||
|
||
The site builds (`prerender-static.sh` here, and `aac-aurasense-web`'s
|
||
`scripts/build-app.sh`) vendor the runtime and copy the parts to `/app/ai/`,
|
||
after `check_packs.py` has matched them to the committed catalogue. Set
|
||
`AAC_AI=none` to build without them (CI does); a deploy of such a build
|
||
removes the model from the site.
|
||
|
||
## In the browser
|
||
|
||
- **Download.** Settings → Sentence helper. The person sees the size and
|
||
whether they are on Wi-Fi before anything starts.
|
||
- **Storage.** The worker and the one WebAssembly build this browser runs are
|
||
stored in the model store (Cache API `aac-models-v1`), with every part of the
|
||
model. Each file is SHA-256 checked before it is kept.
|
||
- **Loading.** The worker streams the parts, one after another, from the model
|
||
store into GPU memory (LiteRT-LM's GPU Artisan backend). The model never sits
|
||
in JavaScript memory whole.
|
||
- **When it runs.** Only where WebGPU has a hardware adapter that is not
|
||
refused (`ai_helper::fit`). Never on the processor.
|
||
- **Checking it.** After download it is loaded once and asked one question.
|
||
The first load compiles the graphics programs, once. A device that cannot run
|
||
it says so then, rather than at the person's first request.
|
||
- **Which model.** Each language is served by the smallest model that passed
|
||
its test. A device keeps one model: the larger one also serves every
|
||
language the smaller one does, so changing to a language that needs it offers
|
||
the larger model in place of the smaller (Settings says so, with its size),
|
||
and never the other way round. The larger model asks for a device with at
|
||
least 8 GB of memory, the smaller one 4 GB (`min_memory_gb`).
|
||
- **Memory.** About 1.8 GB of graphics memory for E2B and 3.0 GB for E4B while
|
||
it is in use. It is let go
|
||
after 10 minutes without a request, and loaded again (seconds) on the next
|
||
one.
|
||
|
||
Measured on 2026-09-30 and 2026-10-01:
|
||
|
||
| | MacBook Pro M5 Max, Chrome 154 | iPad Air 11-inch (M3), Safari 27.0.1 |
|
||
|---|---|---|
|
||
| Runtime build | `litertlm_wasm_internal` (relaxed SIMD + JSPI) | `litertlm_wasm_compat_internal` (JSPI) |
|
||
| Download and check, 2.0 GB | 8 s from a local server | 101 s over Tailscale |
|
||
| First start, once | about 4 s | about 80 s |
|
||
| Later starts | 2.7 s | 5.9 s |
|
||
| One answer (3 sentences or 4 replies) | 0.4–0.7 s | 0.8–1.4 s |
|
||
|
||
## Languages
|
||
|
||
A language is offered only after testing. The test is 20 everyday exchanges
|
||
(`eval/items.json`), and each answer is read for whether one suggestion says
|
||
what the person meant, in good grammar, and how many of the replies are
|
||
natural. A language is ready on a model when at least 15 of 20 sentences are
|
||
usable and the readers count at least 60 of 80 replies good on average; it is
|
||
offered with the smallest model it is ready on. `languages.json` records the
|
||
status, model and a plain note for each tested language.
|
||
|
||
English, Spanish, German and Hindi were read by hand (`eval/results-2026-10-01.json`).
|
||
The other 41 languages were read by two independent model readers and, where
|
||
their verdicts could change the outcome, by a third (Gemma 4 31B, `eval/judge.py`,
|
||
which agreed with the hand reading on 75 of 80 verdicts); a sentence is usable
|
||
when most of its readers say so (`eval/results-2026-10-01-languages.json`). No
|
||
native speaker has read these yet.
|
||
|
||
- **Ready with E2B (17):** Arabic, Danish, English, French, German, Hindi,
|
||
Indonesian, Korean, Malay, Persian, Portuguese, Russian, Spanish, Swedish,
|
||
Turkish, Urdu, Vietnamese.
|
||
- **Ready with E4B (14):** Chinese, Dutch, Gujarati, Hebrew, Italian, Japanese,
|
||
Marathi, Norwegian, Polish, Punjabi, Tagalog, Telugu, Thai, Ukrainian.
|
||
- **Not yet (14):** Amharic, Bengali, Czech, Finnish, Greek, Hausa, Hungarian,
|
||
Kannada, Malayalam, Romanian, Swahili, Tamil, Yoruba, Zulu. Their notes say
|
||
what went wrong.
|
||
|
||
The same few exchanges fail in nearly every language, with both models: "I'm
|
||
frustrated, nobody understands me" comes out as "I don't understand", and "no,
|
||
I just had lunch" loses its reason. They count against every language alike.
|
||
|
||
To add or retest a language:
|
||
|
||
1. Write its 20 exchanges and its examples (two sentence examples, one with
|
||
what the partner said and one without, and two reply examples), in that
|
||
language, as `<code>.json`, and merge them: `python3 eval/merge_languages.py DIR code`.
|
||
It enters the language as not yet tested.
|
||
2. Run the exchanges on each model: `LANGUAGES=1 node eval/run.mjs SITE OUT.json code`.
|
||
3. Read the answers, ideally with a native speaker; `eval/judge.py` is a
|
||
second reader, not the only one.
|
||
4. Set its status, model and note in `languages.json`, and rebuild.
|
||
|
||
The examples are all the model is shown in that language, besides the
|
||
instructions, which say the language's name.
|
||
|
||
## Updating the runtime or the model
|
||
|
||
- **Runtime.** Pin the new `@litert-lm/core` release in `vendor.sh`, bump
|
||
`LITERT_LM_VERSION` and `LITERT_LM_VENDOR_REV` in `ai_helper.rs`, then run
|
||
`vendor.sh` and copy the sizes and hashes it prints into `RUNTIME_FILES` and
|
||
`VENDOR_DIGEST`. The model's `runtime` field in the catalogue must name the
|
||
new runtime. A model made for another runtime is not offered.
|
||
- **Model.** Change its entry in `build_ai.py` `MODELS` (revision, size,
|
||
SHA-256), test the languages again, rebuild and redeploy. People who have the old model see
|
||
"A newer version is available"; the update downloads only the parts that
|
||
changed, and removes the old ones.
|