Selection (access-core; SPEC §5.4, proposed): a prior over the options on the board — usage frequency now, the Laya decision model later — times the person's own switch-timing likelihood, after Nomon (Broderick & MacKay): - press_timing.rs learns a damped kernel-density estimate of each press's offset from the intended highlight, from a population prior scaled to the person's scan interval, and only from selections the person keeps: presses are staged per attempt, dropped when abandoned or undone (including an Undo reached by scanning), and learned only after three newer selections. - selection.rs computes the posterior (late presses explained as late, a 2% lapse term), selects at 99:1 odds with the prior capped so presses must favour the choice 4:1, and otherwise keeps scanning — a row gets up to two extra passes. Without opt-in, or before 10 kept presses, it behaves exactly like the conventional scanner and only measures. - scan_order.rs orders rows, groups and cells by prior mass (fewest expected steps) without moving any; options keep stable position-order ids, and a new order applies from the next pass. dwell.rs weights per-cell dwell from 0.8x to 1.5x within hard bounds. Every adaptation is opt-in and previewable. Conformance C8–C11 (order never moves a position, opt-in required, dwell bounds, selection follows presses): 370/370 with the original 236. Prediction (aac-core): per-language packs — word frequencies, word pairs, sentence starts and a letter model, front-coded and CRC-checked — built reproducibly by tools/lang/build_packs.py from pinned wordfreq 3.2.0 (CC-BY-SA-4.0), FrequencyWords 2018 (CC-BY-SA-4.0) and Common Voice sentences (CC0), into data/lang/ (43 of 45 languages; gu and kn have no open source large enough and report "adaptive only"). PredictionEngine blends a pack with what the person has said (hierarchical Dirichlet smoothing): next word, completion, and letter prediction for letter scanning, deterministically. A per-language pair filter exists; its lists still need curating before the app shows next-word suggestions to children. Co-Authored-By: Claude Opus 5.5 <[email protected]>
34 KiB
Language pack data — licences
Generated by aac/aac-rs/tools/lang/build_packs.py v2; do not edit by hand.
The files in this directory are data, not code, and are not covered by the
Apache-2.0 licence of the reference code around them. Each <code>.<hash>.bin pack is
licensed as stated in its own header and in index.json:
- Creative Commons Attribution-ShareAlike 4.0 International (
CC-BY-SA-4.0, https://creativecommons.org/licenses/by-sa/4.0/):ar,bn,cs,da,de,el,en,es,fa,fi,fr,he,hi,hu,id,it,ja,ko,ml,ms,nb,nl,pl,pt,ro,ru,sv,ta,te,th,tl,tr,uk,ur,vi,zh. - CC0 1.0 Universal (public domain dedication) (
CC0-1.0, https://creativecommons.org/publicdomain/zero/1.0/):am,ha,mr,pa,sw,yo,zu.
A CC BY-SA 4.0 pack is an adaptation (word, word-pair and letter statistics only) of the sources credited below, and is shared under the same licence: if you share it or an adaptation of it, keep the attribution and the licence. ShareAlike applies to the pack files, not to software that loads them.
index.json repeats each pack's licence, attribution and coverage note; it is licensed
CC BY-SA 4.0, as it carries the attributions of CC BY-SA packs. Languages without a
pack (gu, kn) have no data here.
Attribution, per pack
am(CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.ar(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.bn(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.cs(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.da(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.de(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-DE by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.el(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.en(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-US and SUBTLEX-UK by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.es(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.fa(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.fi(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.fr(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.ha(CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.he(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.hi(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.hu(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.id(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.it(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.ja(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.ko(CC-BY-SA-4.0): Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.ml(CC-BY-SA-4.0): Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.mr(CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.ms(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.nb(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.nl(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-NL by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.pa(CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.pl(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.pt(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.ro(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.ru(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.sv(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.sw(CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication.ta(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikisource and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.te(CC-BY-SA-4.0): Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.th(CC-BY-SA-4.0): Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.tl(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.tr(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.uk(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.ur(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.vi(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.yo(CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.zh(CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-CH by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; the word list that comes with the Jieba segmenter, (c) 2013 Sun Junyi, MIT (https://github.com/fxsjy/jieba), whose provenance wordfreq says it does not know. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.zu(CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.