Transaction-Science/open-standards
Watch
1
0
Fork
You've already forked open-standards
 0
Code Issues Pull requests Projects Releases 17 Packages Wiki Activity Actions 47
open-standards/aac/aac-rs/data/lang/LICENSE.md
Transaction Science edcf02b350 feat(access-core, aac-core): Bayesian selection that never moves a cell, and prediction in 43 languages 
Selection (access-core; SPEC §5.4, proposed): a prior over the options on the
board — usage frequency now, the Laya decision model later — times the
person's own switch-timing likelihood, after Nomon (Broderick & MacKay):
- press_timing.rs learns a damped kernel-density estimate of each press's
  offset from the intended highlight, from a population prior scaled to the
  person's scan interval, and only from selections the person keeps: presses
  are staged per attempt, dropped when abandoned or undone (including an Undo
  reached by scanning), and learned only after three newer selections.
- selection.rs computes the posterior (late presses explained as late, a
  2% lapse term), selects at 99:1 odds with the prior capped so presses must
  favour the choice 4:1, and otherwise keeps scanning — a row gets up to two
  extra passes. Without opt-in, or before 10 kept presses, it behaves exactly
  like the conventional scanner and only measures.
- scan_order.rs orders rows, groups and cells by prior mass (fewest expected
  steps) without moving any; options keep stable position-order ids, and a
  new order applies from the next pass. dwell.rs weights per-cell dwell from
  0.8x to 1.5x within hard bounds. Every adaptation is opt-in and previewable.
Conformance C8–C11 (order never moves a position, opt-in required, dwell
bounds, selection follows presses): 370/370 with the original 236.

Prediction (aac-core): per-language packs — word frequencies, word pairs,
sentence starts and a letter model, front-coded and CRC-checked — built
reproducibly by tools/lang/build_packs.py from pinned wordfreq 3.2.0
(CC-BY-SA-4.0), FrequencyWords 2018 (CC-BY-SA-4.0) and Common Voice sentences
(CC0), into data/lang/ (43 of 45 languages; gu and kn have no open source
large enough and report "adaptive only"). PredictionEngine blends a pack with
what the person has said (hierarchical Dirichlet smoothing): next word,
completion, and letter prediction for letter scanning, deterministically.
A per-language pair filter exists; its lists still need curating before the
app shows next-word suggestions to children.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-29 11:28:26 -04:00

65 lines
34 KiB
Markdown
Raw Permalink Blame History

# Language pack data — licences
Generated by `aac/aac-rs/tools/lang/build_packs.py` v2; do not edit by hand.
The files in this directory are **data, not code**, and are **not** covered by the
Apache-2.0 licence of the reference code around them. Each `<code>.<hash>.bin` pack is
licensed as stated in its own header and in `index.json`:
- **Creative Commons Attribution-ShareAlike 4.0 International** (`CC-BY-SA-4.0`, https://creativecommons.org/licenses/by-sa/4.0/): `ar`, `bn`, `cs`, `da`, `de`, `el`, `en`, `es`, `fa`, `fi`, `fr`, `he`, `hi`, `hu`, `id`, `it`, `ja`, `ko`, `ml`, `ms`, `nb`, `nl`, `pl`, `pt`, `ro`, `ru`, `sv`, `ta`, `te`, `th`, `tl`, `tr`, `uk`, `ur`, `vi`, `zh`.
- **CC0 1.0 Universal (public domain dedication)** (`CC0-1.0`, https://creativecommons.org/publicdomain/zero/1.0/): `am`, `ha`, `mr`, `pa`, `sw`, `yo`, `zu`.
A CC BY-SA 4.0 pack is an adaptation (word, word-pair and letter statistics only) of the
sources credited below, and is shared under the same licence: if you share it or an
adaptation of it, keep the attribution and the licence. ShareAlike applies to the pack
files, not to software that loads them.
`index.json` repeats each pack's licence, attribution and coverage note; it is licensed
CC BY-SA 4.0, as it carries the attributions of CC BY-SA packs. Languages without a
pack (`gu`, `kn`) have no data here.
## Attribution, per pack
- `am` (CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.
- `ar` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `bn` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `cs` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `da` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `de` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-DE by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `el` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `en` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-US and SUBTLEX-UK by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `es` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `fa` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `fi` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `fr` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `ha` (CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.
- `he` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `hi` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `hu` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `id` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `it` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `ja` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `ko` (CC-BY-SA-4.0): Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `ml` (CC-BY-SA-4.0): Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `mr` (CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.
- `ms` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `nb` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `nl` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-NL by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `pa` (CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.
- `pl` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `pt` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `ro` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `ru` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `sv` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `sw` (CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication.
- `ta` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikisource and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `te` (CC-BY-SA-4.0): Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `th` (CC-BY-SA-4.0): Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `tl` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `tr` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `uk` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `ur` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `vi` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `yo` (CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.
- `zh` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-CH by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; the word list that comes with the Jieba segmenter, (c) 2013 Sun Junyi, MIT (https://github.com/fxsjy/jieba), whose provenance wordfreq says it does not know. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.
- `zu` (CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.
Reference in a new issue View git blame Copy permalink