open-standards/aac/aac-rs/tools/ai/README.md

# The sentence helper's model

Express (aac-web) can download an on-device language model, on request, and
use it to suggest whole sentences from the words a person chose, or replies to
what was said to them (`crates/aac-web/src/ai_helper.rs`). It runs in the
browser on the graphics chip (WebGPU), in a worker, with Google's LiteRT-LM web
runtime. Nothing typed is sent anywhere, and it works offline once downloaded.

| Piece | Where | Pinned by |
|---|---|---|
| Models: Gemma 4 E2B and E4B, Google AI Edge's web builds (`gemma-4-E2B-it-web.litertlm`, 2,008,432,640 bytes; `gemma-4-E4B-it-web.litertlm`, 2,969,059,328 bytes; Apache-2.0) | `/app/ai/` (this tool's output) | Hugging Face revision, size and SHA-256 in `build_ai.py` `MODELS` |
| Runtime: `@litert-lm/core` 0.17.1 (four WebAssembly builds) and the app's worker | `/app/vendor/litert-lm-<version>-<rev>/` | npm integrity in `third_party/litert-lm/vendor.sh`; each file's size and SHA-256 in `ai_helper.rs` `RUNTIME_FILES` |
| Languages, their status, and the examples the model is shown | `languages.json` → the catalogue | this repository |

## Build and publish

```bash
python3 tools/ai/build_ai.py                    # download the pinned models once (into .cache/), split, write the catalogue
python3 tools/ai/build_ai.py --source ID=FILE   # or use a local copy of a model (its size and SHA-256 are checked)
python3 tools/ai/check_packs.py ../../../aac-ai-packs
```

The output goes beside the open-standards checkout, in `aac-ai-packs/` (outside
git; `$AAC_AI_DIR` or `--out` override). It holds:

- `index.json`: a pointer to the current catalogue, served no-store.
- `index.<sha12>.json`: the catalogue (schema `aac-ai-index/1`). It lists the
  models' parts by size and SHA-256, and each tested language's status, model,
  note and examples. Its committed copy is `data/ai/index.json`.
- `<model-id>/part-NN.<sha12>.bin`: each model file in 64 MiB parts.
  Together the parts are the file, unchanged. Each part is small enough for a
  tablet's model store to hash in memory, and for Cloudflare to cache.
- `LICENSES.md` and the licence text.

The site builds (`prerender-static.sh` here, and `aac-aurasense-web`'s
`scripts/build-app.sh`) vendor the runtime and copy the parts to `/app/ai/`,
after `check_packs.py` has matched them to the committed catalogue. Set
`AAC_AI=none` to build without them (CI does); a deploy of such a build
removes the model from the site.

## In the browser

- **Download.** Settings → Sentence helper. The person sees the size and
  whether they are on Wi-Fi before anything starts.
- **Storage.** The worker and the one WebAssembly build this browser runs are
  stored in the model store (Cache API `aac-models-v1`), with every part of the
  model. Each file is SHA-256 checked before it is kept.
- **Loading.** The worker streams the parts, one after another, from the model
  store into GPU memory (LiteRT-LM's GPU Artisan backend). The model never sits
  in JavaScript memory whole.
- **When it runs.** Only where WebGPU has a hardware adapter that is not
  refused (`ai_helper::fit`). Never on the processor.
- **Checking it.** After download it is loaded once and asked one question.
  The first load compiles the graphics programs, once. A device that cannot run
  it says so then, rather than at the person's first request.
- **Which model.** Each language is served by the smallest model that passed
  its test. A device keeps one model: the larger one also serves every
  language the smaller one does, so changing to a language that needs it offers
  the larger model in place of the smaller (Settings says so, with its size),
  and never the other way round. The larger model asks for a device with at
  least 8 GB of memory, the smaller one 4 GB (`min_memory_gb`).
- **Memory.** About 1.8 GB of graphics memory for E2B and 3.0 GB for E4B while
  it is in use. It is let go
  after 10 minutes without a request, and loaded again (seconds) on the next
  one.

Measured on 2026-09-30 and 2026-10-01:

| | MacBook Pro M5 Max, Chrome 154 | iPad Air 11-inch (M3), Safari 27.0.1 |
|---|---|---|
| Runtime build | `litertlm_wasm_internal` (relaxed SIMD + JSPI) | `litertlm_wasm_compat_internal` (JSPI) |
| Download and check, 2.0 GB | 8 s from a local server | 101 s over Tailscale |
| First start, once | about 4 s | about 80 s |
| Later starts | 2.7 s | 5.9 s |
| One answer (3 sentences or 4 replies) | 0.4–0.7 s | 0.8–1.4 s |

## Languages

A language is offered only after testing. The test is 20 everyday exchanges
(`eval/items.json`), and each answer is read for whether one suggestion says
what the person meant, in good grammar, and how many of the replies are
natural. A language is ready on a model when at least 15 of 20 sentences are
usable and the readers count at least 60 of 80 replies good on average; it is
offered with the smallest model it is ready on. `languages.json` records the
status, model and a plain note for each tested language.

English, Spanish, German and Hindi were read by hand (`eval/results-2026-10-01.json`).
The other 41 languages were read by two independent model readers and, where
their verdicts could change the outcome, by a third (Gemma 4 31B, `eval/judge.py`,
which agreed with the hand reading on 75 of 80 verdicts); a sentence is usable
when most of its readers say so (`eval/results-2026-10-01-languages.json`). No
native speaker has read these yet.

- **Ready with E2B (17):** Arabic, Danish, English, French, German, Hindi,
  Indonesian, Korean, Malay, Persian, Portuguese, Russian, Spanish, Swedish,
  Turkish, Urdu, Vietnamese.
- **Ready with E4B (14):** Chinese, Dutch, Gujarati, Hebrew, Italian, Japanese,
  Marathi, Norwegian, Polish, Punjabi, Tagalog, Telugu, Thai, Ukrainian.
- **Not yet (14):** Amharic, Bengali, Czech, Finnish, Greek, Hausa, Hungarian,
  Kannada, Malayalam, Romanian, Swahili, Tamil, Yoruba, Zulu. Their notes say
  what went wrong.

The same few exchanges fail in nearly every language, with both models: "I'm
frustrated, nobody understands me" comes out as "I don't understand", and "no,
I just had lunch" loses its reason. They count against every language alike.

To add or retest a language:

1. Write its 20 exchanges and its examples (two sentence examples, one with
   what the partner said and one without, and two reply examples), in that
   language, as `<code>.json`, and merge them: `python3 eval/merge_languages.py DIR code`.
   It enters the language as not yet tested.
2. Run the exchanges on each model: `LANGUAGES=1 node eval/run.mjs SITE OUT.json code`.
3. Read the answers, ideally with a native speaker; `eval/judge.py` is a
   second reader, not the only one.
4. Set its status, model and note in `languages.json`, and rebuild.

The examples are all the model is shown in that language, besides the
instructions, which say the language's name.

## Updating the runtime or the model

- **Runtime.** Pin the new `@litert-lm/core` release in `vendor.sh`, bump
  `LITERT_LM_VERSION` and `LITERT_LM_VENDOR_REV` in `ai_helper.rs`, then run
  `vendor.sh` and copy the sizes and hashes it prints into `RUNTIME_FILES` and
  `VENDOR_DIGEST`. The model's `runtime` field in the catalogue must name the
  new runtime. A model made for another runtime is not offered.
- **Model.** Change its entry in `build_ai.py` `MODELS` (revision, size,
  SHA-256), test the languages again, rebuild and redeploy. People who have the old model see
  "A newer version is available"; the update downloads only the parts that
  changed, and removes the old ones.