open-standards/aac/aac-rs/tools/ae/eval
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Transaction Science d7d11fbd9b feat(aac first letters): the model's files, and English with evidence
tools/ae/build_ae.py publishes the abbreviation-expansion model made in the
model library (gemma4-web/abbrev): Qwen3-0.6B-Base (Apache-2.0) fine-tuned
to turn first letters into sentences, INT4 block 16 for WebGPU, every node
on WebGPU. Each file is checked against its pinned size and SHA-256, named
by its contents and kept at most 64 MiB: the one weights file that holds a
single 74 MiB tensor (the word embeddings) is stored as two parts. The
content-named catalogue (aac-ae-index/1, committed as data/ae/index.json)
lists every part, the credits (the model's licence, and the CC BY 4.0
conversations it learned from), and each language's status and evidence;
check_packs.py matches a pack directory to it before a site ships it.

A language is ready only when we measured it on the app's own worker with
the reply meant among the five sentences for at least 50 in 100 of at least
200 replies (tools/ae/eval/run.mjs). English, through the app's worker in
Chrome on WebGPU: 52.8% among five and 32.9% first on 252 Turk Dialogues
replies with what the partner said (56.3% punctuation aside), 85.5% on the
held-out conversations; forgiving slips to a neighbouring key doubles the
hits on replies typed with slips (33.5% -> 66.5% held out) for a loss of
0.5-1.6 points on clean typing. The 50-item check gives 26 first and 36
among five (the model library: 26/37 Chrome, 27/37 WebKit). Every other
language says why not yet.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-02 00:51:00 -04:00
..
run.mjs feat(aac first letters): the model's files, and English with evidence 2026-10-02 00:51:00 -04:00
summary-2026-10-01.json feat(aac first letters): the model's files, and English with evidence 2026-10-02 00:51:00 -04:00