edcf02b350
Selection (access-core; SPEC §5.4, proposed): a prior over the options on the board — usage frequency now, the Laya decision model later — times the person's own switch-timing likelihood, after Nomon (Broderick & MacKay): - press_timing.rs learns a damped kernel-density estimate of each press's offset from the intended highlight, from a population prior scaled to the person's scan interval, and only from selections the person keeps: presses are staged per attempt, dropped when abandoned or undone (including an Undo reached by scanning), and learned only after three newer selections. - selection.rs computes the posterior (late presses explained as late, a 2% lapse term), selects at 99:1 odds with the prior capped so presses must favour the choice 4:1, and otherwise keeps scanning — a row gets up to two extra passes. Without opt-in, or before 10 kept presses, it behaves exactly like the conventional scanner and only measures. - scan_order.rs orders rows, groups and cells by prior mass (fewest expected steps) without moving any; options keep stable position-order ids, and a new order applies from the next pass. dwell.rs weights per-cell dwell from 0.8x to 1.5x within hard bounds. Every adaptation is opt-in and previewable. Conformance C8–C11 (order never moves a position, opt-in required, dwell bounds, selection follows presses): 370/370 with the original 236. Prediction (aac-core): per-language packs — word frequencies, word pairs, sentence starts and a letter model, front-coded and CRC-checked — built reproducibly by tools/lang/build_packs.py from pinned wordfreq 3.2.0 (CC-BY-SA-4.0), FrequencyWords 2018 (CC-BY-SA-4.0) and Common Voice sentences (CC0), into data/lang/ (43 of 45 languages; gu and kn have no open source large enough and report "adaptive only"). PredictionEngine blends a pack with what the person has said (hierarchical Dirichlet smoothing): next word, completion, and letter prediction for letter scanning, deterministically. A per-language pair filter exists; its lists still need curating before the app shows next-word suggestions to children. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1143 lines
50 KiB
Python
1143 lines
50 KiB
Python
#!/usr/bin/env python3.14
|
||
"""Build the per-language prediction packs that Express (aac-core) loads.
|
||
|
||
Usage (from aac/aac-rs):
|
||
|
||
python3.14 tools/lang/build_packs.py # verify lock, build
|
||
python3.14 tools/lang/build_packs.py --update-lock # (re)pin new sources
|
||
python3.14 tools/lang/build_packs.py --offline # cache only, no network
|
||
|
||
Standard library only. Every input is fetched from an immutable URL (a git
|
||
commit, never a branch) and checked against `sources.lock.json` (SHA-256 and
|
||
byte length) before it is used; a mismatch stops the build. With the same lock
|
||
the output is byte-identical: nothing depends on time, locale, hash seeds or
|
||
dictionary order, and every sort has a total order. Normalization and letter
|
||
classes come from Python's Unicode database, so the build also requires the
|
||
Python and Unicode versions below (UNICODE_VERSION, PYTHON_VERSION) and
|
||
records them in every pack; a different interpreter stops the build unless
|
||
--update-lock is given.
|
||
|
||
Output, into aac-rs/data/lang/ (core-owned; the app copies it into its
|
||
public/lang/ when it serves the packs):
|
||
|
||
<code>.<sha256-12>.bin one pack per language (content-addressed name)
|
||
index.json code -> file, bytes, sha256, quality, sources,
|
||
licence, coverage note; plus every language that
|
||
has NO pack and why (SPEC §4.1: say so)
|
||
LICENSE.md the data licence of every pack and of index.json,
|
||
with each pack's attribution — the pack data is
|
||
CC BY-SA 4.0 or CC0 1.0, not the Apache-2.0 of the
|
||
code around it
|
||
|
||
Sources (verified 2026-09-29 against the live repositories):
|
||
|
||
* wordfreq 3.2.0 — https://github.com/rspeer/wordfreq, commit
|
||
912caf64b657478d1dff1138efdc078947d54bb1 (2025-01-04). "Copyright 2022
|
||
Robyn Speer" (NOTICE.md). Code Apache-2.0 (LICENSE.txt); data CC-BY-SA-4.0
|
||
per README §License, with attribution to its upstream corpora. Which text
|
||
domains each language draws on is README's "Sources and supported
|
||
languages" table, copied into WORDFREQ_DOMAINS below (re-read 2026-09-29);
|
||
each pack credits only its own language's domains. Word frequencies are a
|
||
snapshot "through about 2021" (README); the project is sunset. Its lists
|
||
carry at least three independent text domains per language. Its Korean list
|
||
is MeCab morphemes (이, 는, 하 …), not the space-separated words people
|
||
type, so Korean is taken from FrequencyWords instead.
|
||
* FrequencyWords — https://github.com/hermitdave/FrequencyWords, commit
|
||
525f9b560de45753a5ea01069454e72e9aa541c6 (2022-02-07). "Copyright (c) 2016
|
||
Hermit Dave" (LICENSE, MIT, for the code); content CC-BY-SA-4.0 (README
|
||
§License). Counts from OPUS OpenSubtitles 2018. OPUS's own page for that
|
||
release (http://opus.nlpl.eu/OpenSubtitles2018.php) now returns 404
|
||
(2026-09-29), so no licence statement from OPUS could be read; wordfreq's
|
||
licence notice says the data "originates from the OpenSubtitles project …
|
||
and may be used with attribution to OpenSubtitles". The chain of title for
|
||
the FrequencyWords-only languages (ko, ml, te, th) is therefore thinner than
|
||
for wordfreq's; they keep the conservative CC-BY-SA-4.0 label and say so in
|
||
their notes. One source domain (film/TV subtitles): conversational, noisy.
|
||
Used for ko, ml, te, th, and to restore German ß (wordfreq case-folds ß to
|
||
ss).
|
||
* Mozilla Common Voice sentence corpus — https://github.com/common-voice/
|
||
common-voice, commit 4e581f24eee3a58882923752590d084a3e5e9525
|
||
(2026-09-29), server/data/<locale>/*.txt, CC0-1.0 (server/data/LICENSE).
|
||
Sentences written or collected for people to read aloud; some files are
|
||
sentences Mozilla drew from Wikipedia, Wikisource or Europarl and releases
|
||
under the same CC0, and packs that use them say so. Used for word pairs
|
||
(bigrams) and sentence starts in every language it has, and as the only word
|
||
list for sw, am, ha, yo, zu, mr, pa.
|
||
|
||
Checked and not used:
|
||
|
||
* Leipzig Corpora Collection (wortschatz.uni-leipzig.de/en/download): the
|
||
download page is behind an anti-scraper proof-of-work wall (2026-09-29), so
|
||
an automated, pinned build cannot fetch it; its terms were not readable
|
||
there either. Candidate for kn/gu if fetched and licence-checked by hand.
|
||
|
||
What a pack holds (all derived, never raw text):
|
||
|
||
1. Words: up to MAX_WORDS of the most frequent words, each with a cost
|
||
c = round(-100 * log10 p) in centibels (wordfreq's own unit: its lists
|
||
store words in exactly these bins, so for wordfreq languages c is the bin
|
||
index unchanged). Stored sorted by UTF-8 bytes and front-coded, so the
|
||
runtime can binary-search a prefix.
|
||
2. Sentence starts: P(w | start of sentence), top MAX_STARTS.
|
||
3. Word pairs: P(w2 | w1) for w1 in frequency order, top MAX_FOLLOWERS each,
|
||
pairs seen at least MIN_PAIR_COUNT times, both words in the word table.
|
||
4. Letters: a character model of order 2 (P(next | two previous symbols),
|
||
with a word-boundary symbol) estimated from the word table weighted by
|
||
word frequency — the classic prefix-based letter predictor (MacKenzie et
|
||
al., "LetterWise", UIST 2001), used by the runtime only when the typed
|
||
prefix matches no known word.
|
||
|
||
Binary layout (little-endian; varint = unsigned LEB128):
|
||
|
||
"AACLPK" u16=1 magic, format version
|
||
u32 meta_len, meta (UTF-8 JSON, sorted keys) source, licence, attribution…
|
||
words: varint n; n × (varint lcp_bytes, varint suffix_len, suffix, varint cost)
|
||
starts: varint n; n × (varint word_index, varint cost)
|
||
pairs: varint contexts; each (varint Δw1, varint k, k × (varint Δw2, varint cost))
|
||
letters: varint alphabet; alphabet × varint Δcodepoint
|
||
order0: varint k, k × (varint sym, varint cost)
|
||
order1: varint contexts; each (varint Δs1, varint k, k × (sym, cost))
|
||
order2: varint contexts; each (varint Δkey, varint k, k × (sym, cost))
|
||
key = s1 * (alphabet + 1) + s2; symbol 0 = word boundary
|
||
u32 CRC-32 (IEEE) of everything above
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import argparse
|
||
import gzip
|
||
import hashlib
|
||
import json
|
||
import math
|
||
import os
|
||
import re
|
||
import struct
|
||
import sys
|
||
import unicodedata
|
||
import urllib.request
|
||
import zlib
|
||
from collections import Counter, defaultdict
|
||
from pathlib import Path
|
||
|
||
BUILDER_VERSION = "2"
|
||
FORMAT_VERSION = 1
|
||
# The Unicode database (unicodedata.normalize / category) and interpreter the
|
||
# packs were built with; either changing can change the output silently.
|
||
UNICODE_VERSION = "16.0.0"
|
||
PYTHON_VERSION = (3, 14)
|
||
|
||
HERE = Path(__file__).resolve().parent
|
||
AAC_RS = HERE.parent.parent
|
||
DEFAULT_OUT = AAC_RS / "data" / "lang"
|
||
PAIR_FILTER_DIR = HERE / "pair_filter"
|
||
LOCK_PATH = HERE / "sources.lock.json"
|
||
I18N_RS = AAC_RS / "crates" / "aac-core" / "src" / "i18n.rs"
|
||
|
||
# ─── Pinned sources ─────────────────────────────────────────────────────────
|
||
|
||
WORDFREQ_COMMIT = "912caf64b657478d1dff1138efdc078947d54bb1"
|
||
WORDFREQ_RAW = f"https://raw.githubusercontent.com/rspeer/wordfreq/{WORDFREQ_COMMIT}/wordfreq/data"
|
||
FW_COMMIT = "525f9b560de45753a5ea01069454e72e9aa541c6"
|
||
FW_RAW = f"https://raw.githubusercontent.com/hermitdave/FrequencyWords/{FW_COMMIT}/content/2018"
|
||
CV_COMMIT = "4e581f24eee3a58882923752590d084a3e5e9525"
|
||
CV_RAW = f"https://raw.githubusercontent.com/common-voice/common-voice/{CV_COMMIT}"
|
||
CV_TREE = f"https://api.github.com/repos/common-voice/common-voice/git/trees/{CV_COMMIT}?recursive=1"
|
||
|
||
SOURCE_INFO = {
|
||
"wordfreq": {
|
||
"name": "wordfreq 3.2.0 (Robyn Speer)",
|
||
"url": f"https://github.com/rspeer/wordfreq/tree/{WORDFREQ_COMMIT}",
|
||
"licence": "CC-BY-SA-4.0",
|
||
},
|
||
"frequencywords": {
|
||
"name": "FrequencyWords 2018 (Hermit Dave)",
|
||
"url": f"https://github.com/hermitdave/FrequencyWords/tree/{FW_COMMIT}",
|
||
"licence": "CC-BY-SA-4.0",
|
||
},
|
||
"commonvoice": {
|
||
"name": "Mozilla Common Voice sentence corpus",
|
||
"url": f"https://github.com/common-voice/common-voice/tree/{CV_COMMIT}/server/data",
|
||
"licence": "CC0-1.0",
|
||
},
|
||
}
|
||
|
||
# wordfreq README at WORDFREQ_COMMIT, "Sources and supported languages": the
|
||
# text domains each language's list combines (a "Yes" in its row), for the
|
||
# languages built from wordfreq here. Parsed from the README fetched
|
||
# 2026-09-29; "fil" is wordfreq's code for Tagalog.
|
||
WORDFREQ_DOMAINS = {
|
||
"ar": ["wikipedia", "subtitles", "news", "web", "twitter"],
|
||
"bn": ["wikipedia", "subtitles", "news", "web", "twitter"],
|
||
"cs": ["wikipedia", "subtitles", "news", "web", "twitter"],
|
||
"da": ["wikipedia", "subtitles", "web", "twitter"],
|
||
"de": ["wikipedia", "subtitles", "news", "books", "web", "twitter", "reddit"],
|
||
"el": ["wikipedia", "subtitles", "web", "twitter"],
|
||
"en": ["wikipedia", "subtitles", "news", "books", "web", "twitter", "reddit"],
|
||
"es": ["wikipedia", "subtitles", "news", "books", "web", "twitter", "reddit"],
|
||
"fa": ["wikipedia", "subtitles", "web", "twitter"],
|
||
"fi": ["wikipedia", "subtitles", "news", "web", "twitter", "reddit"],
|
||
"fil": ["wikipedia", "subtitles", "web"],
|
||
"fr": ["wikipedia", "subtitles", "news", "books", "web", "twitter", "reddit"],
|
||
"he": ["wikipedia", "subtitles", "books", "web", "twitter"],
|
||
"hi": ["wikipedia", "web", "twitter", "reddit"],
|
||
"hu": ["wikipedia", "subtitles", "web", "twitter"],
|
||
"id": ["wikipedia", "subtitles", "twitter"],
|
||
"it": ["wikipedia", "subtitles", "news", "books", "web", "twitter", "reddit"],
|
||
"ja": ["wikipedia", "subtitles", "web", "twitter", "reddit"],
|
||
"ms": ["wikipedia", "subtitles", "twitter"],
|
||
"nb": ["wikipedia", "subtitles", "web", "twitter", "reddit"],
|
||
"nl": ["wikipedia", "subtitles", "news", "web", "twitter"],
|
||
"pl": ["wikipedia", "subtitles", "news", "web", "twitter", "reddit"],
|
||
"pt": ["wikipedia", "subtitles", "news", "web", "twitter"],
|
||
"ro": ["wikipedia", "subtitles", "web"],
|
||
"ru": ["wikipedia", "subtitles", "news", "books", "twitter"],
|
||
"sv": ["wikipedia", "subtitles", "web", "twitter", "reddit"],
|
||
"ta": ["wikipedia", "web", "twitter"],
|
||
"tr": ["wikipedia", "subtitles", "web", "twitter"],
|
||
"uk": ["wikipedia", "subtitles", "web", "twitter", "reddit"],
|
||
"ur": ["wikipedia", "web", "twitter"],
|
||
"vi": ["wikipedia", "subtitles", "web"],
|
||
"zh": ["wikipedia", "subtitles", "news", "books", "web", "twitter", "misc"],
|
||
}
|
||
# README §License: SUBTLEX lists in wordfreq, which "must credit the SUBTLEX
|
||
# authors" and keep it "clear that SUBTLEX is freely available data".
|
||
SUBTLEX = {"en": "SUBTLEX-US and SUBTLEX-UK", "zh": "SUBTLEX-CH", "de": "SUBTLEX-DE", "nl": "SUBTLEX-NL"}
|
||
|
||
|
||
def wordfreq_attribution(wf_code: str) -> str:
|
||
"""Credit wordfreq and exactly the domains this language's list uses."""
|
||
domains = WORDFREQ_DOMAINS[wf_code]
|
||
code = "tl" if wf_code == "fil" else wf_code
|
||
credit = {
|
||
"wikipedia": "Wikipedia (https://www.wikipedia.org)",
|
||
"subtitles": ("subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), "
|
||
"which originates from OpenSubtitles (http://www.opensubtitles.org)"
|
||
+ (f", and {SUBTLEX[code]} by Marc Brysbaert et al. "
|
||
"(http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely "
|
||
"available data)" if code in SUBTLEX else "")),
|
||
"news": "news from NewsCrawl 2014 and GlobalVoices",
|
||
"books": "Google Books Ngrams 2012 (http://books.google.com/ngrams)",
|
||
"web": "web text from OSCAR",
|
||
"twitter": "word statistics from Twitter",
|
||
"reddit": "Reddit comments",
|
||
"misc": ("the word list that comes with the Jieba segmenter, (c) 2013 Sun Junyi, MIT "
|
||
"(https://github.com/fxsjy/jieba), whose provenance wordfreq says it does not know"),
|
||
}
|
||
return ("Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer "
|
||
"(https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. "
|
||
"For this language it combines " + "; ".join(credit[d] for d in domains) + ". "
|
||
"wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds "
|
||
"Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu).")
|
||
|
||
|
||
FREQUENCYWORDS_ATTRIBUTION = (
|
||
"Word counts from FrequencyWords, (c) 2016 Hermit Dave "
|
||
"(https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from "
|
||
"OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org)."
|
||
)
|
||
|
||
|
||
def commonvoice_attribution(cv_files: list[str]) -> str:
|
||
"""Credit Common Voice, and say where Mozilla drew bulk sentences from."""
|
||
drawn = []
|
||
names = [f.rsplit("/", 1)[-1] for f in cv_files]
|
||
if any(n.startswith("wiki") and "wikisource" not in n for n in names):
|
||
drawn.append("Wikipedia")
|
||
if any("wikisource" in n for n in names):
|
||
drawn.append("Wikisource")
|
||
if any("europarl" in n for n in names):
|
||
drawn.append("Europarl")
|
||
origin = (f", including sentences Mozilla drew from {' and '.join(drawn)} and releases under the "
|
||
"same dedication" if drawn else "")
|
||
return ("Sentences from the Mozilla Common Voice sentence corpus "
|
||
"(https://github.com/common-voice/common-voice), dedicated to the public domain under "
|
||
f"CC0 1.0{origin}.")
|
||
|
||
|
||
LICENCE_URL = {
|
||
"CC-BY-SA-4.0": "https://creativecommons.org/licenses/by-sa/4.0/",
|
||
"CC0-1.0": "https://creativecommons.org/publicdomain/zero/1.0/",
|
||
}
|
||
|
||
# ─── Languages ──────────────────────────────────────────────────────────────
|
||
|
||
# The 45 site languages, in i18n.rs order (checked against it at start-up).
|
||
LANGS = [
|
||
"en", "es", "fr", "de", "pt", "it", "nl", "zh", "ja", "ko", "ar", "he", "ur",
|
||
"fa", "hi", "bn", "ta", "te", "ml", "mr", "kn", "gu", "pa", "th", "vi", "id",
|
||
"ms", "tl", "ru", "pl", "uk", "ro", "cs", "sv", "da", "nb", "fi", "sw", "am",
|
||
"ha", "yo", "zu", "tr", "el", "hu",
|
||
]
|
||
|
||
# Where each language's word list comes from.
|
||
WORDFREQ_CODE = {c: c for c in [
|
||
"en", "es", "fr", "de", "pt", "it", "nl", "zh", "ja", "ar", "he", "ur", "fa",
|
||
"hi", "bn", "ta", "vi", "id", "ms", "ru", "pl", "uk", "ro", "cs", "sv", "da",
|
||
"nb", "fi", "tr", "el", "hu",
|
||
]} | {"tl": "fil"}
|
||
FW_FILE = {"ko": "ko/ko_50k.txt", "ml": "ml/ml_50k.txt", "te": "te/te_full.txt", "th": "th/th_50k.txt"}
|
||
CV_ONLY = {"sw", "am", "ha", "yo", "zu", "mr", "pa", "kn", "gu"}
|
||
CV_LOCALE = {"zh": "zh-CN", "sv": "sv-SE", "nb": "nb-NO", "pa": "pa-IN"}
|
||
|
||
# Languages written without spaces: word pairs need dictionary segmentation.
|
||
SPACELESS = {"zh", "ja", "th"}
|
||
|
||
# Elision (wordfreq's tokenizer rule, from UAX #29's optional WB5a): one or two
|
||
# letters, an apostrophe, then a vowel or h, split after the apostrophe —
|
||
# French "l'eau" is "l'" + "eau". wordfreq stores the particle without its
|
||
# apostrophe ("l"), which is not something a person would choose to say, so
|
||
# the particles below get it back.
|
||
ELISION = {
|
||
"fr": {"c", "d", "j", "l", "m", "n", "s", "t", "qu"},
|
||
"it": {"c", "d", "l", "m", "n", "s", "t", "v"},
|
||
}
|
||
ELISION_VOWELS = "aeiouhàâäéèêëîïôöùûüÿœæy"
|
||
|
||
# Display forms where the lower-cased word is not how anyone writes it.
|
||
DISPLAY = {
|
||
"en": {"i": "I", "i'm": "I'm", "i'll": "I'll", "i've": "I've", "i'd": "I'd"},
|
||
}
|
||
|
||
SCRIPT = {
|
||
**{c: "Latn" for c in ["en", "es", "fr", "de", "pt", "it", "nl", "vi", "id", "ms",
|
||
"tl", "pl", "ro", "cs", "sv", "da", "nb", "fi", "sw", "ha",
|
||
"yo", "zu", "tr", "hu"]},
|
||
"zh": "Hani", "ja": "Jpan", "ko": "Hang", "ar": "Arab", "ur": "Arab", "fa": "Arab",
|
||
"he": "Hebr", "hi": "Deva", "mr": "Deva", "bn": "Beng", "ta": "Taml", "te": "Telu",
|
||
"ml": "Mlym", "kn": "Knda", "gu": "Gujr", "pa": "Guru", "th": "Thai", "ru": "Cyrl",
|
||
"uk": "Cyrl", "el": "Grek", "am": "Ethi",
|
||
}
|
||
|
||
SCRIPT_RANGES = {
|
||
"Latn": [(0x41, 0x5A), (0x61, 0x7A), (0xC0, 0x24F), (0x250, 0x2AF), (0x1E00, 0x1EFF),
|
||
(0x300, 0x36F)],
|
||
"Cyrl": [(0x400, 0x52F), (0x300, 0x36F)],
|
||
"Grek": [(0x370, 0x3FF), (0x1F00, 0x1FFF), (0x300, 0x36F)],
|
||
"Arab": [(0x600, 0x6FF), (0x750, 0x77F), (0x8A0, 0x8FF), (0xFB50, 0xFDFF),
|
||
(0xFE70, 0xFEFF), (0x200C, 0x200D)],
|
||
"Hebr": [(0x590, 0x5FF), (0xFB1D, 0xFB4F)],
|
||
"Deva": [(0x900, 0x97F), (0xA8E0, 0xA8FF), (0x200C, 0x200D)],
|
||
"Beng": [(0x980, 0x9FF), (0x200C, 0x200D)],
|
||
"Guru": [(0xA00, 0xA7F), (0x200C, 0x200D)],
|
||
"Gujr": [(0xA80, 0xAFF), (0x200C, 0x200D)],
|
||
"Taml": [(0xB80, 0xBFF), (0x200C, 0x200D)],
|
||
"Telu": [(0xC00, 0xC7F), (0x200C, 0x200D)],
|
||
"Knda": [(0xC80, 0xCFF), (0x200C, 0x200D)],
|
||
"Mlym": [(0xD00, 0xD7F), (0x200C, 0x200D)],
|
||
"Thai": [(0xE00, 0xE7F)],
|
||
"Ethi": [(0x1200, 0x139F), (0x2D80, 0x2DDF)],
|
||
"Hang": [(0xAC00, 0xD7AF), (0x1100, 0x11FF), (0x3130, 0x318F)],
|
||
"Hani": [(0x4E00, 0x9FFF), (0x3400, 0x4DBF), (0xF900, 0xFAFF)],
|
||
"Jpan": [(0x4E00, 0x9FFF), (0x3400, 0x4DBF), (0x3040, 0x309F), (0x30A0, 0x30FF),
|
||
(0x3005, 0x3005), (0x303B, 0x303B), (0x3006, 0x3006)],
|
||
}
|
||
|
||
# ─── Budgets (bytes are of the encoded section) ─────────────────────────────
|
||
|
||
MAX_WORDS = 16000
|
||
WORD_BYTES = 200_000 # before front-coding, which saves about a third
|
||
MAX_STARTS = 64
|
||
MAX_FOLLOWERS = 12
|
||
MIN_PAIR_COUNT = 2
|
||
PAIR_BYTES = 90_000
|
||
MAX_ALPHABET = 400
|
||
MAX_LETTER_FOLLOWERS = 16
|
||
LETTER_BYTES = 45_000
|
||
CV_BYTES_PER_LANG = 4_000_000
|
||
MIN_TOKENS = 5_000 # fewer corpus tokens than this: no pack
|
||
SPARSE_TOKENS = 100_000 # fewer than this: labelled "sparse"
|
||
FEW_PAIRS = 1_000 # fewer word pairs than this: say so in the notes
|
||
MAX_SEG_LEN = 8 # longest dictionary word tried when segmenting
|
||
|
||
# ─── Fetching and pinning ───────────────────────────────────────────────────
|
||
|
||
|
||
class Fetcher:
|
||
"""Downloads into a cache and checks every byte string against the lock."""
|
||
|
||
def __init__(self, cache: Path, offline: bool, update_lock: bool):
|
||
self.cache = cache
|
||
self.offline = offline
|
||
self.update_lock = update_lock
|
||
lock = json.loads(LOCK_PATH.read_text()) if LOCK_PATH.exists() else {}
|
||
self.lock: dict[str, dict] = lock.get("files", {})
|
||
# Common Voice files per locale ([path, size]) at the pinned commit.
|
||
# Listed once from the GitHub tree API with --update-lock; builds then
|
||
# use this listing, because an API response is not a pinned artifact.
|
||
self.cv_listing: dict[str, list] = lock.get("commonvoice_listing", {})
|
||
self.used: dict[str, dict] = {}
|
||
cache.mkdir(parents=True, exist_ok=True)
|
||
|
||
def get(self, url: str, max_bytes: int | None = None) -> bytes:
|
||
key = url if max_bytes is None else f"{url}#bytes=0-{max_bytes - 1}"
|
||
path = self.cache / hashlib.sha256(key.encode()).hexdigest()
|
||
if path.exists():
|
||
data = path.read_bytes()
|
||
else:
|
||
if self.offline:
|
||
sys.exit(f"offline and not cached: {key}")
|
||
req = urllib.request.Request(url, headers={"User-Agent": "aac-langpack-builder/1"})
|
||
if max_bytes is not None:
|
||
req.add_header("Range", f"bytes=0-{max_bytes - 1}")
|
||
with urllib.request.urlopen(req, timeout=120) as r:
|
||
data = r.read()
|
||
path.write_bytes(data)
|
||
digest = hashlib.sha256(data).hexdigest()
|
||
pinned = self.lock.get(key)
|
||
if pinned is None or pinned["sha256"] != digest or pinned["bytes"] != len(data):
|
||
if not self.update_lock:
|
||
sys.exit(f"source does not match sources.lock.json: {key}\n"
|
||
f" got sha256 {digest} ({len(data)} bytes); "
|
||
f"rerun with --update-lock only if the change is intended")
|
||
self.used[key] = {"sha256": digest, "bytes": len(data)}
|
||
return data
|
||
|
||
def write_lock(self) -> None:
|
||
lock = {"files": dict(sorted(self.used.items())),
|
||
"commonvoice_listing": dict(sorted(self.cv_listing.items()))}
|
||
LOCK_PATH.write_text(json.dumps(lock, indent=1, sort_keys=True, ensure_ascii=False) + "\n")
|
||
|
||
def list_commonvoice(self) -> None:
|
||
"""Refresh the Common Voice file listing (only with --update-lock)."""
|
||
req = urllib.request.Request(CV_TREE, headers={"User-Agent": "aac-langpack-builder/1"})
|
||
with urllib.request.urlopen(req, timeout=120) as r:
|
||
tree = json.loads(r.read())
|
||
if tree.get("truncated"):
|
||
sys.exit("Common Voice tree listing is truncated")
|
||
listing: dict[str, list] = defaultdict(list)
|
||
for e in tree["tree"]:
|
||
parts = e["path"].split("/")
|
||
if (e["type"] == "blob" and len(parts) == 4 and parts[:2] == ["server", "data"]
|
||
and parts[3].endswith(".txt")):
|
||
listing[parts[2]].append([e["path"], e["size"]])
|
||
wanted = {CV_LOCALE.get(c, c) for c in LANGS}
|
||
self.cv_listing = {k: sorted(v) for k, v in listing.items() if k in wanted}
|
||
|
||
|
||
# ─── Normalization and tokens (mirrors wordfreq's preprocess_text) ──────────
|
||
|
||
|
||
def normal_form(code: str) -> str:
|
||
"""wordfreq's choice (NFC for cased alphabets, NFKC otherwise), except Thai.
|
||
|
||
NFKC decomposes Thai SARA AM (ำ, U+0E33) into NIKHAHIT + SARA AA, so a
|
||
suggestion would be inserted in a form keyboards never produce. Thai comes
|
||
from FrequencyWords, not wordfreq, so it can use NFC.
|
||
"""
|
||
return "NFC" if SCRIPT[code] in ("Latn", "Grek", "Cyrl", "Thai") else "NFKC"
|
||
|
||
|
||
def normalize(text: str, code: str) -> str:
|
||
"""NFC/NFKC, abjad mark removal, lower-casing, Turkish/Romanian letters.
|
||
|
||
Lower-casing is Unicode default lower-casing (Python str.lower, which is
|
||
what Rust's str::to_lowercase does), not case folding: case folding turns
|
||
German ß into ss and Greek final ς into σ, which the person would see.
|
||
"""
|
||
text = unicodedata.normalize(normal_form(code), text)
|
||
text = text.replace("’", "'").replace("ʼ", "'")
|
||
if code == "th":
|
||
text = text.replace("ํา", "ำ") # the two spellings of SARA AM
|
||
if SCRIPT[code] in ("Arab", "Hebr"):
|
||
text = "".join(ch for ch in text
|
||
if unicodedata.category(ch) != "Mn" and ch != "ـ")
|
||
if code == "tr":
|
||
text = text.replace("İ", "i").replace("I", "ı")
|
||
text = text.lower()
|
||
if code == "tr":
|
||
text = text.replace("ș", "ş").replace("ț", "ţ")
|
||
elif code == "ro":
|
||
text = text.replace("ş", "ș").replace("ţ", "ț")
|
||
return text
|
||
|
||
|
||
def in_script(ch: str, code: str) -> bool:
|
||
cp = ord(ch)
|
||
return any(lo <= cp <= hi for lo, hi in SCRIPT_RANGES[SCRIPT[code]])
|
||
|
||
|
||
def is_word(tok: str, code: str) -> bool:
|
||
"""A word is letters of the language's script; an apostrophe only inside."""
|
||
if not tok or tok.startswith("'"):
|
||
return False
|
||
if tok.endswith("'") and not (code in ELISION and tok[:-1] in ELISION[code]):
|
||
return False
|
||
letters = 0
|
||
for ch in tok:
|
||
if ch == "'":
|
||
continue
|
||
if not in_script(ch, code):
|
||
return False
|
||
if unicodedata.category(ch)[0] == "L":
|
||
letters += 1
|
||
return letters > 0
|
||
|
||
|
||
def tokens(text: str, code: str | None = None) -> list[str]:
|
||
"""Runs of letters, marks and digits, with apostrophes inside a run.
|
||
|
||
With `code` in ELISION, a run like "l'eau" is split into "l'" and "eau".
|
||
"""
|
||
out: list[str] = []
|
||
for tok in _runs(text):
|
||
if code in ELISION:
|
||
m = re.fullmatch(rf"(\w\w?')([{ELISION_VOWELS}].*)", tok)
|
||
if m:
|
||
out.extend([m.group(1), m.group(2)])
|
||
continue
|
||
out.append(tok)
|
||
return out
|
||
|
||
|
||
def _runs(text: str) -> list[str]:
|
||
out, cur = [], []
|
||
for ch in text:
|
||
cat = unicodedata.category(ch)
|
||
if cat[0] in "LMN" or ch in "" or (ch == "'" and cur):
|
||
cur.append(ch)
|
||
else:
|
||
if cur:
|
||
out.append("".join(cur).strip("'"))
|
||
cur = []
|
||
if cur:
|
||
out.append("".join(cur).strip("'"))
|
||
return [t for t in out if t]
|
||
|
||
|
||
def thai_mojibake(tok: str) -> bool:
|
||
"""UTF-8 Thai decoded as TIS-620: เ + (ธ|น) + a byte 0xA1–0xBF (ก…ฟ).
|
||
|
||
Thai UTF-8 is E0 B8 xx / E0 B9 xx; read as TIS-620, E0 is เ, B8 is ธ, B9
|
||
is น and 0xA1–0xBF are ก…ฟ. Genuine words like เธอ or เน้น do not match,
|
||
because อ and ้ lie outside ก…ฟ.
|
||
"""
|
||
return bool(re.search("เ[ธน]([ก-ฟ]|$)", tok))
|
||
|
||
|
||
# ─── Word lists ─────────────────────────────────────────────────────────────
|
||
|
||
|
||
def msgpack_unpack(b: bytes):
|
||
"""The subset of MessagePack that wordfreq's cBpack files use."""
|
||
pos = 0
|
||
|
||
def take(n):
|
||
nonlocal pos
|
||
v = b[pos:pos + n]
|
||
pos += n
|
||
return v
|
||
|
||
def rd():
|
||
t = take(1)[0]
|
||
if t <= 0x7F:
|
||
return t
|
||
if 0x80 <= t <= 0x8F:
|
||
return {rd(): rd() for _ in range(t & 0x0F)}
|
||
if 0x90 <= t <= 0x9F:
|
||
return [rd() for _ in range(t & 0x0F)]
|
||
if 0xA0 <= t <= 0xBF:
|
||
return take(t & 0x1F).decode()
|
||
if t == 0xC0:
|
||
return None
|
||
if t in (0xC2, 0xC3):
|
||
return t == 0xC3
|
||
if t >= 0xE0:
|
||
return t - 256
|
||
size = {0xCC: 1, 0xCD: 2, 0xCE: 4, 0xD9: 1, 0xDA: 2, 0xDB: 4,
|
||
0xDC: 2, 0xDD: 4, 0xDE: 2, 0xDF: 4}.get(t)
|
||
if size is None:
|
||
raise ValueError(f"msgpack type {t:#x} not supported")
|
||
n = int.from_bytes(take(size), "big")
|
||
if t in (0xCC, 0xCD, 0xCE):
|
||
return n
|
||
if t in (0xD9, 0xDA, 0xDB):
|
||
return take(n).decode()
|
||
if t in (0xDC, 0xDD):
|
||
return [rd() for _ in range(n)]
|
||
return {rd(): rd() for _ in range(n)}
|
||
|
||
return rd()
|
||
|
||
|
||
def wordfreq_costs(fetch: Fetcher, code: str) -> dict[str, int]:
|
||
"""wordfreq cBpack: element i (i ≥ 1) lists the words of frequency 10^(-i/100)."""
|
||
raw = fetch.get(f"{WORDFREQ_RAW}/small_{WORDFREQ_CODE[code]}.msgpack.gz")
|
||
data = msgpack_unpack(gzip.decompress(raw))
|
||
header = data[0]
|
||
if header.get("format") != "cB" or header.get("version") != 1:
|
||
raise ValueError(f"unexpected wordfreq header {header}")
|
||
costs: dict[str, int] = {}
|
||
for cost, words in enumerate(data[1:], start=1):
|
||
for w in words:
|
||
# wordfreq case-folds; undo the two folds a reader would notice.
|
||
if code == "el" and w.endswith("σ"):
|
||
w = w[:-1] + "ς"
|
||
if code in ELISION and w in ELISION[code]:
|
||
w += "'"
|
||
if is_word(w, code) and w not in costs:
|
||
costs[w] = cost
|
||
return costs
|
||
|
||
|
||
def frequencywords_counts(fetch: Fetcher, rel: str, code: str) -> Counter:
|
||
text = fetch.get(f"{FW_RAW}/{rel}").decode("utf-8", errors="replace")
|
||
counts: Counter = Counter()
|
||
for line in text.splitlines():
|
||
parts = line.rsplit(" ", 1)
|
||
if len(parts) != 2 or not parts[1].isdigit():
|
||
continue
|
||
w = normalize(parts[0], code)
|
||
if code == "th" and (thai_mojibake(w) or len(w) == 1):
|
||
continue # mojibake, or a lone Thai letter (subtitle line-break debris)
|
||
if is_word(w, code):
|
||
counts[w] += int(parts[1])
|
||
return counts
|
||
|
||
|
||
def restore_eszett(costs: dict[str, int], fetch: Fetcher) -> int:
|
||
"""wordfreq folds ß to ss; take the spelling subtitles use most often."""
|
||
fw = frequencywords_counts(fetch, "de/de_50k.txt", "de")
|
||
best: dict[str, tuple[int, str]] = {}
|
||
for w, c in fw.items():
|
||
folded = w.replace("ß", "ss")
|
||
if (c, w) > best.get(folded, (-1, "")):
|
||
best[folded] = (c, w)
|
||
restored = 0
|
||
for w in list(costs):
|
||
if "ss" in w and w in best and best[w][1] != w:
|
||
target = best[w][1]
|
||
if target not in costs:
|
||
costs[target] = costs.pop(w)
|
||
restored += 1
|
||
return restored
|
||
|
||
|
||
def counts_to_costs(counts: Counter) -> dict[str, int]:
|
||
total = sum(counts.values())
|
||
return {w: max(1, round(-100 * math.log10(c / total))) for w, c in counts.items()}
|
||
|
||
|
||
# ─── Common Voice sentences ─────────────────────────────────────────────────
|
||
|
||
|
||
def cv_sentences(fetch: Fetcher, code: str) -> tuple[list[str], list[str]]:
|
||
"""Up to CV_BYTES_PER_LANG of sentences, community files before wiki dumps."""
|
||
locale = CV_LOCALE.get(code, code)
|
||
files = [tuple(f) for f in fetch.cv_listing.get(locale, [])]
|
||
bulk = re.compile(r"/(wiki|europarl)[^/]*$")
|
||
files.sort(key=lambda f: (bool(bulk.search(f[0])), f[0]))
|
||
remaining = CV_BYTES_PER_LANG
|
||
lines, used = [], []
|
||
for path, size in files:
|
||
if remaining <= 0:
|
||
break
|
||
url = f"{CV_RAW}/{path}"
|
||
if size <= remaining:
|
||
data = fetch.get(url)
|
||
else:
|
||
data = fetch.get(url, max_bytes=remaining)
|
||
data = data[: data.rfind(b"\n") + 1]
|
||
remaining -= len(data)
|
||
used.append(path[len("server/data/"):])
|
||
lines.extend(data.decode("utf-8", errors="replace").splitlines())
|
||
return lines, used
|
||
|
||
|
||
def segment(run: str, lexicon: set[str]) -> list[str | None]:
|
||
"""Forward maximum matching against the pack's own word list.
|
||
|
||
The dictionary baseline for Chinese word segmentation (e.g. Wong & Chan,
|
||
COLING 1996): at each position take the longest dictionary word. A
|
||
character that starts no dictionary word becomes None, which breaks the
|
||
chain of word pairs. Approximate — it is only used to count word pairs.
|
||
"""
|
||
out: list[str | None] = []
|
||
i = 0
|
||
while i < len(run):
|
||
for j in range(min(len(run), i + MAX_SEG_LEN), i, -1):
|
||
if run[i:j] in lexicon:
|
||
out.append(run[i:j])
|
||
i = j
|
||
break
|
||
else:
|
||
out.append(None)
|
||
i += 1
|
||
return out
|
||
|
||
|
||
def sentence_words(line: str, code: str, lexicon: set[str] | None) -> list[str]:
|
||
words = []
|
||
for tok in tokens(normalize(line, code), code):
|
||
if not is_word(tok, code):
|
||
words.append(None) # a number or foreign word breaks the chain
|
||
continue
|
||
if code in SPACELESS and lexicon is not None:
|
||
words.extend(segment(tok, lexicon))
|
||
else:
|
||
words.append(tok)
|
||
return words
|
||
|
||
|
||
# ─── Encoding ───────────────────────────────────────────────────────────────
|
||
|
||
|
||
def varint(n: int) -> bytes:
|
||
if n < 0:
|
||
raise ValueError(n)
|
||
out = bytearray()
|
||
while True:
|
||
b = n & 0x7F
|
||
n >>= 7
|
||
if n:
|
||
out.append(b | 0x80)
|
||
else:
|
||
out.append(b)
|
||
return bytes(out)
|
||
|
||
|
||
def lcp_chars(a: str, b: str) -> int:
|
||
"""Longest common prefix in bytes, cut back to a character boundary."""
|
||
n = 0
|
||
for x, y in zip(a, b):
|
||
if x != y:
|
||
break
|
||
n += len(x.encode())
|
||
return n
|
||
|
||
|
||
def cost_of(p: float) -> int:
|
||
return max(0, min(65535, round(-100 * math.log10(p))))
|
||
|
||
|
||
# ─── One pack ───────────────────────────────────────────────────────────────
|
||
|
||
|
||
def compact_files(s: dict) -> dict:
|
||
"""Name up to 8 input files; beyond that, their count (the lock has them all)."""
|
||
files = s.get("files", [])
|
||
if len(files) <= 8:
|
||
return s
|
||
return {**{k: v for k, v in s.items() if k != "files"},
|
||
"files": files[:8] + [f"… and {len(files) - 8} more (see sources.lock.json)"]}
|
||
|
||
|
||
|
||
def pair_filter(code: str) -> set[str]:
|
||
"""Words never offered as a next word or a sentence start.
|
||
|
||
`tools/lang/pair_filter/<code>.txt`, one word per line (`#` comments),
|
||
normalized like the pack. The words stay in the word table, so they still
|
||
complete when the person types them — the person chooses the words; the
|
||
pack only stops proposing them unprompted. The lists are curated by hand
|
||
(see pair_filter/README.md); a language without a file withholds nothing.
|
||
"""
|
||
path = PAIR_FILTER_DIR / f"{code}.txt"
|
||
if not path.exists():
|
||
return set()
|
||
words = set()
|
||
for line in path.read_text(encoding="utf-8").splitlines():
|
||
line = line.split("#", 1)[0].strip()
|
||
if line:
|
||
words.add(normalize(line, code))
|
||
return words
|
||
|
||
|
||
def build_pack(code: str, fetch: Fetcher) -> tuple[bytes | None, dict]:
|
||
notes: list[str] = []
|
||
sources: list[dict] = []
|
||
corpus_lines, cv_files = cv_sentences(fetch, code)
|
||
|
||
# 1. Word costs.
|
||
if code in WORDFREQ_CODE:
|
||
costs = wordfreq_costs(fetch, code)
|
||
quality = "multi_source"
|
||
unigram_source = "wordfreq"
|
||
unigram_tokens = None
|
||
if code == "de":
|
||
n = restore_eszett(costs, fetch)
|
||
notes.append(f"ß restored in {n} words from FrequencyWords (wordfreq folds ß to ss)")
|
||
sources.append({"role": "spelling", "source": "frequencywords",
|
||
"files": ["de/de_50k.txt"]})
|
||
elif code in FW_FILE:
|
||
counts = frequencywords_counts(fetch, FW_FILE[code], code)
|
||
costs = counts_to_costs(counts)
|
||
unigram_source = "frequencywords"
|
||
unigram_tokens = sum(counts.values())
|
||
quality = "single_source" if unigram_tokens >= SPARSE_TOKENS else "sparse"
|
||
notes.append("word list from film and TV subtitles (OPUS OpenSubtitles 2018), which is "
|
||
"published with a request for attribution to OpenSubtitles rather than an "
|
||
"explicit licence; labelled CC BY-SA 4.0 as FrequencyWords' content is")
|
||
else:
|
||
counts = Counter(w for line in corpus_lines
|
||
for w in sentence_words(line, code, None) if w)
|
||
unigram_tokens = sum(counts.values())
|
||
not_yet = ("candidate open sources (e.g. Wikipedia dumps, CC BY-SA) are not yet "
|
||
"integrated")
|
||
if not cv_files:
|
||
return None, {"reason": "none of the sources this builder uses (wordfreq 3.2.0, "
|
||
"FrequencyWords 2018, Common Voice) covers this language; "
|
||
+ not_yet}
|
||
if unigram_tokens < MIN_TOKENS:
|
||
return None, {"reason": f"not in wordfreq 3.2.0 or FrequencyWords 2018, and Common "
|
||
f"Voice has only {unigram_tokens} tokens (< {MIN_TOKENS}), too "
|
||
f"few to seed prediction; {not_yet}"}
|
||
costs = counts_to_costs(counts)
|
||
unigram_source = "commonvoice"
|
||
quality = "single_source" if unigram_tokens >= SPARSE_TOKENS else "sparse"
|
||
sources.insert(0, {"role": "words", "source": unigram_source,
|
||
**({"files": [FW_FILE[code]]} if unigram_source == "frequencywords" else {}),
|
||
**({"files": cv_files} if unigram_source == "commonvoice" else {}),
|
||
**({"files": [f"small_{WORDFREQ_CODE[code]}.msgpack.gz"]}
|
||
if unigram_source == "wordfreq" else {}),
|
||
**({"tokens": unigram_tokens} if unigram_tokens else {})})
|
||
|
||
# 2. Choose the word table: most frequent first, then within the byte budget.
|
||
ranked = sorted(costs.items(), key=lambda kv: (kv[1], kv[0].encode()))
|
||
chosen: list[tuple[str, int]] = []
|
||
budget = 0
|
||
for w, c in ranked:
|
||
size = len(w.encode()) + 4
|
||
if len(chosen) >= MAX_WORDS or budget + size > WORD_BYTES:
|
||
break
|
||
chosen.append((w, c))
|
||
budget += size
|
||
words = sorted(chosen, key=lambda wc: wc[0].encode())
|
||
index = {w: i for i, (w, _) in enumerate(words)}
|
||
lexicon = set(index)
|
||
|
||
# 3. Word pairs and sentence starts from Common Voice.
|
||
pair_counts: Counter = Counter()
|
||
left_counts: Counter = Counter()
|
||
start_counts: Counter = Counter()
|
||
corpus_tokens = 0
|
||
for line in corpus_lines:
|
||
ws = sentence_words(line, code, lexicon)
|
||
corpus_tokens += sum(1 for w in ws if w)
|
||
if ws and ws[0] in index:
|
||
start_counts[ws[0]] += 1
|
||
for a, b in zip(ws, ws[1:]):
|
||
if a is None or b is None:
|
||
continue
|
||
left_counts[a] += 1
|
||
if a in index and b in index:
|
||
pair_counts[(a, b)] += 1
|
||
if cv_files:
|
||
sources.append({"role": "pairs", "source": "commonvoice", "files": cv_files,
|
||
"tokens": corpus_tokens})
|
||
notes.append("word pairs and sentence starts come from sentences written for reading "
|
||
"aloud (much of it news and encyclopedia text), not from conversation")
|
||
else:
|
||
notes.append("no Common Voice sentences: no word pairs or sentence starts")
|
||
if code in SPACELESS:
|
||
notes.append("written without spaces: word pairs were counted after dictionary "
|
||
"(maximum-match) segmentation, which is approximate")
|
||
|
||
withheld = pair_filter(code)
|
||
if withheld:
|
||
notes.append(f"{len(withheld)} words are never suggested as a next word or sentence start "
|
||
f"(tools/lang/pair_filter/{code}.txt); they still complete when typed")
|
||
starts_total = sum(start_counts.values())
|
||
starts = sorted(((index[w], cost_of(c / starts_total)) for w, c in start_counts.items()
|
||
if w not in withheld),
|
||
key=lambda ic: (ic[1], ic[0]))[:MAX_STARTS]
|
||
starts.sort()
|
||
|
||
followers: dict[str, list[tuple[int, int]]] = defaultdict(list)
|
||
for (a, b), c in pair_counts.items():
|
||
if c >= MIN_PAIR_COUNT and b not in withheld:
|
||
followers[a].append((index[b], cost_of(c / left_counts[a])))
|
||
pairs: list[tuple[int, list[tuple[int, int]]]] = []
|
||
pair_bytes = 0
|
||
for w, _ in ranked:
|
||
if w not in followers or w not in index:
|
||
continue
|
||
fs = sorted(followers[w], key=lambda ic: (ic[1], ic[0]))[:MAX_FOLLOWERS]
|
||
size = 4 + 4 * len(fs)
|
||
if pair_bytes + size > PAIR_BYTES:
|
||
break
|
||
pairs.append((index[w], sorted(fs)))
|
||
pair_bytes += size
|
||
pairs.sort()
|
||
shipped_pairs = sum(len(f) for _, f in pairs)
|
||
if cv_files and shipped_pairs < FEW_PAIRS:
|
||
# Counted as shipped, after MAX_FOLLOWERS and the byte budget.
|
||
notes.append(f"few word pairs ({shipped_pairs}): next-word suggestions come mostly from "
|
||
"word frequency")
|
||
|
||
# 4. Letters: order-2 character model over the word table.
|
||
probs = [(w, 10 ** (-c / 100)) for w, c in words]
|
||
zp = sum(p for _, p in probs)
|
||
mass: Counter = Counter()
|
||
for w, p in probs:
|
||
for ch in w:
|
||
mass[ch] += p / zp
|
||
alphabet = sorted(sorted(mass, key=lambda ch: (-mass[ch], ord(ch)))[:MAX_ALPHABET], key=ord)
|
||
sym = {ch: i + 1 for i, ch in enumerate(alphabet)} # 0 = word boundary
|
||
o0: Counter = Counter()
|
||
o1: dict[int, Counter] = defaultdict(Counter)
|
||
o2: dict[int, Counter] = defaultdict(Counter)
|
||
base = len(alphabet) + 1
|
||
for w, p in probs:
|
||
if any(ch not in sym for ch in w):
|
||
continue
|
||
seq = [0] + [sym[ch] for ch in w] + [0]
|
||
for i in range(1, len(seq)):
|
||
o0[seq[i]] += p
|
||
o1[seq[i - 1]][seq[i]] += p
|
||
if i >= 2:
|
||
o2[seq[i - 2] * base + seq[i - 1]][seq[i]] += p
|
||
else:
|
||
o2[0 * base + 0][seq[i]] += p # start of word: context (boundary, boundary)
|
||
|
||
def top(dist: Counter) -> list[tuple[int, int]]:
|
||
total = sum(dist.values())
|
||
best = sorted(dist.items(), key=lambda kv: (-kv[1], kv[0]))[:MAX_LETTER_FOLLOWERS]
|
||
return sorted((s, cost_of(v / total)) for s, v in best)
|
||
|
||
order0 = top(o0)
|
||
order1 = sorted((s, top(d)) for s, d in o1.items())
|
||
letter_bytes = 8 * len(alphabet) + sum(8 + 4 * len(f) for _, f in order1)
|
||
order2 = []
|
||
for key, d in sorted(o2.items(), key=lambda kv: (-sum(kv[1].values()), kv[0])):
|
||
fs = top(d)
|
||
size = 4 + 4 * len(fs)
|
||
if letter_bytes + size > LETTER_BYTES:
|
||
break
|
||
order2.append((key, fs))
|
||
letter_bytes += size
|
||
order2.sort()
|
||
|
||
# 5. Metadata: what this is, where it came from, how far to trust it.
|
||
licences = sorted({SOURCE_INFO[s["source"]]["licence"] for s in sources})
|
||
licence = "CC-BY-SA-4.0" if "CC-BY-SA-4.0" in licences else "CC0-1.0"
|
||
used_sources = []
|
||
for s in sorted({s["source"] for s in sources}, key=["wordfreq", "frequencywords",
|
||
"commonvoice"].index):
|
||
used_sources.append(s)
|
||
credit = {
|
||
"wordfreq": lambda: wordfreq_attribution(WORDFREQ_CODE[code]),
|
||
"frequencywords": lambda: FREQUENCYWORDS_ATTRIBUTION,
|
||
"commonvoice": lambda: commonvoice_attribution(cv_files),
|
||
}
|
||
attribution = " ".join(credit[s]() for s in used_sources)
|
||
if licence == "CC-BY-SA-4.0":
|
||
attribution += (" This pack is an adaptation (word, word-pair and letter statistics "
|
||
"only) and is shared under CC BY-SA 4.0.")
|
||
token_note = ""
|
||
if quality == "multi_source":
|
||
coverage = ("Word frequencies from wordfreq, which combines at least three kinds of "
|
||
"text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.")
|
||
elif unigram_source == "frequencywords":
|
||
token_note = f"{unigram_tokens:,} words of film and TV subtitles"
|
||
coverage = f"Word frequencies from one source: {token_note} (OpenSubtitles 2018)."
|
||
else:
|
||
token_note = f"{unigram_tokens:,} words of sentences written for people to read aloud"
|
||
coverage = f"Word frequencies from one small source: {token_note} (Common Voice)."
|
||
if quality == "sparse":
|
||
coverage += " Few words: expect rough suggestions until it learns this person's words."
|
||
meta = {
|
||
"format": "aac-langpack",
|
||
"version": FORMAT_VERSION,
|
||
"builder": f"aac/aac-rs/tools/lang/build_packs.py v{BUILDER_VERSION}",
|
||
"unicode": UNICODE_VERSION,
|
||
"lang": code,
|
||
"quality": quality,
|
||
"coverage_note": coverage,
|
||
"notes": notes,
|
||
"licence": licence,
|
||
"licence_url": LICENCE_URL[licence],
|
||
"attribution": attribution,
|
||
"sources": [
|
||
{**compact_files(s), "name": SOURCE_INFO[s["source"]]["name"],
|
||
"url": SOURCE_INFO[s["source"]]["url"], "licence": SOURCE_INFO[s["source"]]["licence"]}
|
||
for s in sources
|
||
],
|
||
"display": DISPLAY.get(code, {}),
|
||
"normalization": {
|
||
"form": normal_form(code),
|
||
"lowercase": True,
|
||
"remove_marks": SCRIPT[code] in ("Arab", "Hebr"),
|
||
"dotless_i": code == "tr",
|
||
"diacritics_under": {"tr": "cedillas", "ro": "commas"}.get(code),
|
||
"elision": code in ELISION,
|
||
},
|
||
"segmentation": "dictionary" if code in SPACELESS else "spaces",
|
||
"counts": {"words": len(words), "starts": len(starts), "pair_contexts": len(pairs),
|
||
"pairs": sum(len(f) for _, f in pairs), "alphabet": len(alphabet),
|
||
"letter_contexts": len(order1) + len(order2)},
|
||
}
|
||
|
||
# 6. Encode.
|
||
out = bytearray(b"AACLPK")
|
||
out += struct.pack("<H", FORMAT_VERSION)
|
||
mb = json.dumps(meta, sort_keys=True, ensure_ascii=False, separators=(",", ":")).encode()
|
||
out += struct.pack("<I", len(mb)) + mb
|
||
out += varint(len(words))
|
||
prev_s = ""
|
||
for w, c in words:
|
||
wb = w.encode()
|
||
lcp = lcp_chars(prev_s, w)
|
||
out += varint(lcp) + varint(len(wb) - lcp) + wb[lcp:] + varint(c)
|
||
prev_s = w
|
||
out += varint(len(starts))
|
||
for i, c in starts:
|
||
out += varint(i) + varint(c)
|
||
out += varint(len(pairs))
|
||
last = 0
|
||
for w1, fs in pairs:
|
||
out += varint(w1 - last) + varint(len(fs))
|
||
last = w1
|
||
l2 = 0
|
||
for w2, c in fs:
|
||
out += varint(w2 - l2) + varint(c)
|
||
l2 = w2
|
||
out += varint(len(alphabet))
|
||
lastcp = 0
|
||
for ch in alphabet:
|
||
out += varint(ord(ch) - lastcp)
|
||
lastcp = ord(ch)
|
||
out += varint(len(order0))
|
||
for s, c in order0:
|
||
out += varint(s) + varint(c)
|
||
for table in (order1, order2):
|
||
out += varint(len(table))
|
||
lastk = 0
|
||
for k, fs in table:
|
||
out += varint(k - lastk) + varint(len(fs))
|
||
lastk = k
|
||
for s, c in fs:
|
||
out += varint(s) + varint(c)
|
||
out += struct.pack("<I", zlib.crc32(bytes(out)) & 0xFFFFFFFF)
|
||
return bytes(out), meta
|
||
|
||
|
||
# ─── Main ───────────────────────────────────────────────────────────────────
|
||
|
||
|
||
def licence_file(packs: dict[str, dict], missing: dict[str, dict]) -> str:
|
||
"""LICENSE.md for the output directory: the data licence of every file."""
|
||
by = defaultdict(list)
|
||
for code, e in sorted(packs.items()):
|
||
by[e["licence"]].append(code)
|
||
lines = [
|
||
"# Language pack data — licences",
|
||
"",
|
||
f"Generated by `aac/aac-rs/tools/lang/build_packs.py` v{BUILDER_VERSION}; do not edit by hand.",
|
||
"",
|
||
"The files in this directory are **data, not code**, and are **not** covered by the",
|
||
"Apache-2.0 licence of the reference code around them. Each `<code>.<hash>.bin` pack is",
|
||
"licensed as stated in its own header and in `index.json`:",
|
||
"",
|
||
]
|
||
names = {"CC-BY-SA-4.0": "Creative Commons Attribution-ShareAlike 4.0 International",
|
||
"CC0-1.0": "CC0 1.0 Universal (public domain dedication)"}
|
||
for lic in sorted(by):
|
||
lines.append(f"- **{names[lic]}** (`{lic}`, {LICENCE_URL[lic]}): "
|
||
+ ", ".join(f"`{c}`" for c in by[lic]) + ".")
|
||
lines += [
|
||
"",
|
||
"A CC BY-SA 4.0 pack is an adaptation (word, word-pair and letter statistics only) of the",
|
||
"sources credited below, and is shared under the same licence: if you share it or an",
|
||
"adaptation of it, keep the attribution and the licence. ShareAlike applies to the pack",
|
||
"files, not to software that loads them.",
|
||
"",
|
||
"`index.json` repeats each pack's licence, attribution and coverage note; it is licensed",
|
||
"CC BY-SA 4.0, as it carries the attributions of CC BY-SA packs. Languages without a",
|
||
"pack (" + (", ".join(f"`{c}`" for c in sorted(missing)) or "none") + ") have no data here.",
|
||
"",
|
||
"## Attribution, per pack",
|
||
"",
|
||
]
|
||
for code, e in sorted(packs.items()):
|
||
lines.append(f"- `{code}` ({e['licence']}): {e['attribution']}")
|
||
return "\n".join(lines) + "\n"
|
||
|
||
|
||
def check_language_list() -> None:
|
||
src = I18N_RS.read_text()
|
||
codes = re.findall(r'lang\("([a-z]{2,3})"', src)
|
||
if codes != LANGS:
|
||
sys.exit(f"LANGS differs from i18n.rs supported_languages(): {codes}")
|
||
|
||
|
||
def main() -> None:
|
||
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
|
||
ap.add_argument("--out", type=Path, default=DEFAULT_OUT)
|
||
ap.add_argument("--cache", type=Path,
|
||
default=Path(os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache"))
|
||
/ "aac-langpacks")
|
||
ap.add_argument("--offline", action="store_true")
|
||
ap.add_argument("--update-lock", action="store_true")
|
||
ap.add_argument("--only", nargs="*", help="build only these codes (index is not written)")
|
||
args = ap.parse_args()
|
||
|
||
check_language_list()
|
||
if (unicodedata.unidata_version != UNICODE_VERSION or sys.version_info[:2] != PYTHON_VERSION) \
|
||
and not args.update_lock:
|
||
sys.exit(f"built with Unicode {UNICODE_VERSION} on Python {PYTHON_VERSION[0]}.{PYTHON_VERSION[1]}; "
|
||
f"this is Unicode {unicodedata.unidata_version} on Python "
|
||
f"{sys.version_info[0]}.{sys.version_info[1]}, which can change the packs. Use "
|
||
f"python{PYTHON_VERSION[0]}.{PYTHON_VERSION[1]}, or update UNICODE_VERSION and "
|
||
f"PYTHON_VERSION on purpose with --update-lock")
|
||
fetch = Fetcher(args.cache, args.offline, args.update_lock)
|
||
if args.update_lock:
|
||
fetch.list_commonvoice()
|
||
elif not fetch.cv_listing:
|
||
sys.exit("sources.lock.json has no Common Voice listing; run with --update-lock")
|
||
|
||
args.out.mkdir(parents=True, exist_ok=True)
|
||
langs = args.only or LANGS
|
||
packs: dict[str, dict] = {}
|
||
missing: dict[str, dict] = {}
|
||
for code in langs:
|
||
data, meta = build_pack(code, fetch)
|
||
if data is None:
|
||
missing[code] = meta
|
||
print(f"{code}: no pack — {meta['reason']}")
|
||
continue
|
||
digest = hashlib.sha256(data).hexdigest()
|
||
name = f"{code}.{digest[:12]}.bin"
|
||
for old in args.out.glob(f"{code}.*.bin"):
|
||
if old.name != name:
|
||
old.unlink()
|
||
(args.out / name).write_bytes(data)
|
||
packs[code] = {
|
||
"file": name,
|
||
"bytes": len(data),
|
||
"sha256": digest,
|
||
"quality": meta["quality"],
|
||
"licence": meta["licence"],
|
||
"licence_url": meta["licence_url"],
|
||
"sources": sorted({s["name"] for s in meta["sources"]}),
|
||
"coverage_note": meta["coverage_note"],
|
||
"notes": meta["notes"],
|
||
"counts": meta["counts"],
|
||
"attribution": meta["attribution"],
|
||
}
|
||
c = meta["counts"]
|
||
print(f"{code}: {name} {len(data):>7,} B {meta['quality']:<13} words={c['words']} "
|
||
f"pairs={c['pairs']} starts={c['starts']} alphabet={c['alphabet']}")
|
||
|
||
if not args.only:
|
||
index = {
|
||
"format": "aac-langpack-index",
|
||
"version": FORMAT_VERSION,
|
||
"builder": f"aac/aac-rs/tools/lang/build_packs.py v{BUILDER_VERSION}",
|
||
"languages": len(LANGS),
|
||
"unicode": UNICODE_VERSION,
|
||
"packs": packs,
|
||
"missing": missing,
|
||
"fallback": ("A language without a pack predicts only from what this person has "
|
||
"chosen to say (adaptive model); it starts empty."),
|
||
}
|
||
(args.out / "index.json").write_text(
|
||
json.dumps(index, indent=1, sort_keys=True, ensure_ascii=False) + "\n")
|
||
(args.out / "LICENSE.md").write_text(licence_file(packs, missing))
|
||
if args.update_lock:
|
||
fetch.write_lock()
|
||
print(f"\n{len(packs)} packs, {len(missing)} languages without a pack: "
|
||
f"{', '.join(sorted(missing)) or 'none'}")
|
||
|
||
|
||
if __name__ == "__main__":
|
||
main()
|