Transaction-Science/open-standards
Watch
1
0
Fork
You've already forked open-standards
 0
Code Issues Pull requests Projects Releases 15 Packages Wiki Activity Actions
open-standards/aac/aac-rs/data/lang/index.json
Transaction Science edcf02b350 feat(access-core, aac-core): Bayesian selection that never moves a cell, and prediction in 43 languages 
Selection (access-core; SPEC §5.4, proposed): a prior over the options on the
board — usage frequency now, the Laya decision model later — times the
person's own switch-timing likelihood, after Nomon (Broderick & MacKay):
- press_timing.rs learns a damped kernel-density estimate of each press's
  offset from the intended highlight, from a population prior scaled to the
  person's scan interval, and only from selections the person keeps: presses
  are staged per attempt, dropped when abandoned or undone (including an Undo
  reached by scanning), and learned only after three newer selections.
- selection.rs computes the posterior (late presses explained as late, a
  2% lapse term), selects at 99:1 odds with the prior capped so presses must
  favour the choice 4:1, and otherwise keeps scanning — a row gets up to two
  extra passes. Without opt-in, or before 10 kept presses, it behaves exactly
  like the conventional scanner and only measures.
- scan_order.rs orders rows, groups and cells by prior mass (fewest expected
  steps) without moving any; options keep stable position-order ids, and a
  new order applies from the next pass. dwell.rs weights per-cell dwell from
  0.8x to 1.5x within hard bounds. Every adaptation is opt-in and previewable.
Conformance C8–C11 (order never moves a position, opt-in required, dwell
bounds, selection follows presses): 370/370 with the original 236.

Prediction (aac-core): per-language packs — word frequencies, word pairs,
sentence starts and a letter model, front-coded and CRC-checked — built
reproducibly by tools/lang/build_packs.py from pinned wordfreq 3.2.0
(CC-BY-SA-4.0), FrequencyWords 2018 (CC-BY-SA-4.0) and Common Voice sentences
(CC0), into data/lang/ (43 of 45 languages; gu and kn have no open source
large enough and report "adaptive only"). PredictionEngine blends a pack with
what the person has said (hierarchical Dirichlet smoothing): next word,
completion, and letter prediction for letter scanning, deterministically.
A per-language pair filter exists; its lists still need curating before the
app shows next-word suggestions to children.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-29 11:28:26 -04:00

1103 lines
71 KiB
JSON
Raw Permalink Blame History

{
"builder": "aac/aac-rs/tools/lang/build_packs.py v2",
"fallback": "A language without a pack predicts only from what this person has chosen to say (adaptive model); it starts empty.",
"format": "aac-langpack-index",
"languages": 45,
"missing": {
"gu": {
"reason": "none of the sources this builder uses (wordfreq 3.2.0, FrequencyWords 2018, Common Voice) covers this language; candidate open sources (e.g. Wikipedia dumps, CC BY-SA) are not yet integrated"
},
"kn": {
"reason": "not in wordfreq 3.2.0 or FrequencyWords 2018, and Common Voice has only 410 tokens (< 5000), too few to seed prediction; candidate open sources (e.g. Wikipedia dumps, CC BY-SA) are not yet integrated"
}
},
"packs": {
"am": {
"attribution": "Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.",
"bytes": 140062,
"counts": {
"alphabet": 242,
"letter_contexts": 1139,
"pair_contexts": 664,
"pairs": 941,
"starts": 64,
"words": 9486
},
"coverage_note": "Word frequencies from one small source: 20,617 words of sentences written for people to read aloud (Common Voice). Few words: expect rough suggestions until it learns this person's words.",
"file": "am.74f61d8a6e61.bin",
"licence": "CC0-1.0",
"licence_url": "https://creativecommons.org/publicdomain/zero/1.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation",
"few word pairs (941): next-word suggestions come mostly from word frequency"
],
"quality": "sparse",
"sha256": "74f61d8a6e61e16f7badfac8e81ee65608e94e817794bb63415828c09c30095f",
"sources": [
"Mozilla Common Voice sentence corpus"
]
},
"ar": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 176376,
"counts": {
"alphabet": 37,
"letter_contexts": 920,
"pair_contexts": 3909,
"pairs": 11828,
"starts": 64,
"words": 13811
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "ar.098abfc3c3b1.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "098abfc3c3b1b508cfeeaa3f2f11adf899e080ee012fb76c40ceb8a76fa14895",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"bn": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 160151,
"counts": {
"alphabet": 68,
"letter_contexts": 1470,
"pair_contexts": 3115,
"pairs": 9765,
"starts": 64,
"words": 9066
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "bn.14b0cd893d3d.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "14b0cd893d3d51e2d28bb3df933d0d82c1f70e3cd9c5e8bea543b96793b44924",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"cs": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 195564,
"counts": {
"alphabet": 43,
"letter_contexts": 1034,
"pair_contexts": 4795,
"pairs": 17320,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "cs.5d921a9283a5.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "5d921a9283a583d7fcc5c0de7e82dec232f6f266e822bb7292a92e5eb6697426",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"da": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 146113,
"counts": {
"alphabet": 35,
"letter_contexts": 748,
"pair_contexts": 2160,
"pairs": 6056,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "da.cd126aba8373.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "cd126aba8373164266fb94f9160aaf2556f12d331553b1f39efc5c2022075290",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"de": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-DE by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 196085,
"counts": {
"alphabet": 36,
"letter_contexts": 774,
"pair_contexts": 5166,
"pairs": 17193,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "de.d02ace47b59d.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"ß restored in 346 words from FrequencyWords (wordfreq folds ß to ss)",
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "d02ace47b59d1996dc091cfaebd71332b3489dbd114141455629c4b82e3b5770",
"sources": [
"FrequencyWords 2018 (Hermit Dave)",
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"el": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 131473,
"counts": {
"alphabet": 47,
"letter_contexts": 850,
"pair_contexts": 702,
"pairs": 2020,
"starts": 64,
"words": 10758
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "el.3c330d840643.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "3c330d840643a2ba94162c1dbcbad50f90664b038803089b2d6c7af38ac60f6b",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"en": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-US and SUBTLEX-UK by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 190440,
"counts": {
"alphabet": 29,
"letter_contexts": 636,
"pair_contexts": 3755,
"pairs": 18744,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "en.3230df972cd2.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "3230df972cd2df9edce1154fd19f502b6498ac3731eeb19875b5bc487ef713c0",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"es": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 191188,
"counts": {
"alphabet": 37,
"letter_contexts": 751,
"pair_contexts": 4173,
"pairs": 18327,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "es.fd3bcbe54fe1.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "fd3bcbe54fe18479d29b8c26e41f9d0b333adfe3569ebb119f76efd1131ce41e",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"fa": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 189234,
"counts": {
"alphabet": 43,
"letter_contexts": 1056,
"pair_contexts": 4501,
"pairs": 14689,
"starts": 64,
"words": 14075
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "fa.b1b03c209cf3.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "b1b03c209cf3544075843d32ebca59937a7bf5f313642509dd28344f60e98740",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"fi": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 125466,
"counts": {
"alphabet": 31,
"letter_contexts": 575,
"pair_contexts": 470,
"pairs": 966,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "fi.e83ef968627f.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation",
"few word pairs (966): next-word suggestions come mostly from word frequency"
],
"quality": "multi_source",
"sha256": "e83ef968627f099df9c326eefe1ea0bd1df19539dfaf26857e5bfc549eadcfc6",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"fr": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 187220,
"counts": {
"alphabet": 40,
"letter_contexts": 793,
"pair_contexts": 4729,
"pairs": 17128,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "fr.8f2d24509859.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "8f2d24509859faaab7a37d02417c95256cd775124f7c04f1de50e007d2ee3484",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"ha": {
"attribution": "Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.",
"bytes": 54067,
"counts": {
"alphabet": 32,
"letter_contexts": 520,
"pair_contexts": 899,
"pairs": 2699,
"starts": 64,
"words": 5131
},
"coverage_note": "Word frequencies from one small source: 43,220 words of sentences written for people to read aloud (Common Voice). Few words: expect rough suggestions until it learns this person's words.",
"file": "ha.99281e749b8e.bin",
"licence": "CC0-1.0",
"licence_url": "https://creativecommons.org/publicdomain/zero/1.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "sparse",
"sha256": "99281e749b8e56086389d20257c700b89dbab3faf4942d644b8190378998c221",
"sources": [
"Mozilla Common Voice sentence corpus"
]
},
"he": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 128074,
"counts": {
"alphabet": 29,
"letter_contexts": 620,
"pair_contexts": 136,
"pairs": 234,
"starts": 64,
"words": 14391
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "he.fa0916fbf556.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation",
"few word pairs (234): next-word suggestions come mostly from word frequency"
],
"quality": "multi_source",
"sha256": "fa0916fbf556a0c350fe67814cf4bbe60105906c4f4ea5a345238321a34031d4",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"hi": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 193904,
"counts": {
"alphabet": 63,
"letter_contexts": 1325,
"pair_contexts": 4504,
"pairs": 15295,
"starts": 64,
"words": 10103
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "hi.b4db31f77cc7.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "b4db31f77cc729e1d13a1afd0e85939c4498d114813f6f7344732c749f8c17f0",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"hu": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 190021,
"counts": {
"alphabet": 39,
"letter_contexts": 972,
"pair_contexts": 5261,
"pairs": 15238,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "hu.9010a939df5d.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "9010a939df5d9d9e9777f67d0ba4724a82b58ad6764630d38996f1bf35c4a355",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"id": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 134672,
"counts": {
"alphabet": 29,
"letter_contexts": 613,
"pair_contexts": 871,
"pairs": 2572,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "id.c8567644a23d.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "c8567644a23dfb124077384fb3ec24ee3e1d6a9d50e75ac0b439f80ae3ee901e",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"it": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 187839,
"counts": {
"alphabet": 38,
"letter_contexts": 604,
"pair_contexts": 3847,
"pairs": 18650,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "it.b929f258b871.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "b929f258b871204d55d8c87af96561ae49a9b5d40a13d1d4dc56a1d01021666d",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"ja": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 197384,
"counts": {
"alphabet": 400,
"letter_contexts": 1044,
"pair_contexts": 3694,
"pairs": 10228,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "ja.61c4f4f89b3a.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation",
"written without spaces: word pairs were counted after dictionary (maximum-match) segmentation, which is approximate"
],
"quality": "multi_source",
"sha256": "61c4f4f89b3a80ba23ef37b96f08fbd4abf85b3be640bdd4a47d313ff8f0f9b5",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"ko": {
"attribution": "Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 160916,
"counts": {
"alphabet": 400,
"letter_contexts": 895,
"pair_contexts": 630,
"pairs": 1193,
"starts": 64,
"words": 15528
},
"coverage_note": "Word frequencies from one source: 5,161,872 words of film and TV subtitles (OpenSubtitles 2018).",
"file": "ko.6d410ea6cd24.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word list from film and TV subtitles (OPUS OpenSubtitles 2018), which is published with a request for attribution to OpenSubtitles rather than an explicit licence; labelled CC BY-SA 4.0 as FrequencyWords' content is",
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "single_source",
"sha256": "6d410ea6cd24729eff9987fbfc5b007c76532f0a76b69d97c51c13b479781699",
"sources": [
"FrequencyWords 2018 (Hermit Dave)",
"Mozilla Common Voice sentence corpus"
]
},
"ml": {
"attribution": "Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 121039,
"counts": {
"alphabet": 68,
"letter_contexts": 1445,
"pair_contexts": 161,
"pairs": 267,
"starts": 64,
"words": 7470
},
"coverage_note": "Word frequencies from one source: 1,427,673 words of film and TV subtitles (OpenSubtitles 2018).",
"file": "ml.399f6eab576e.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word list from film and TV subtitles (OPUS OpenSubtitles 2018), which is published with a request for attribution to OpenSubtitles rather than an explicit licence; labelled CC BY-SA 4.0 as FrequencyWords' content is",
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation",
"few word pairs (267): next-word suggestions come mostly from word frequency"
],
"quality": "single_source",
"sha256": "399f6eab576ea55aa594654c011d5489449969ac7f87812c73b894d9b16a77a1",
"sources": [
"FrequencyWords 2018 (Hermit Dave)",
"Mozilla Common Voice sentence corpus"
]
},
"mr": {
"attribution": "Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.",
"bytes": 145359,
"counts": {
"alphabet": 71,
"letter_contexts": 1635,
"pair_contexts": 1586,
"pairs": 2687,
"starts": 64,
"words": 8123
},
"coverage_note": "Word frequencies from one small source: 53,520 words of sentences written for people to read aloud (Common Voice). Few words: expect rough suggestions until it learns this person's words.",
"file": "mr.86ce463387d8.bin",
"licence": "CC0-1.0",
"licence_url": "https://creativecommons.org/publicdomain/zero/1.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "sparse",
"sha256": "86ce463387d805590eb52752bf98cbc08ebf6b2b811485c46d7e62ced26bfbad",
"sources": [
"Mozilla Common Voice sentence corpus"
]
},
"ms": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 122356,
"counts": {
"alphabet": 31,
"letter_contexts": 626,
"pair_contexts": 6,
"pairs": 8,
"starts": 39,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "ms.55e1499e38f1.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation",
"few word pairs (8): next-word suggestions come mostly from word frequency"
],
"quality": "multi_source",
"sha256": "55e1499e38f1146a45197d6c5015537712c6f40fcb546e9748c2b5f676de54b9",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"nb": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 125211,
"counts": {
"alphabet": 37,
"letter_contexts": 711,
"pair_contexts": 398,
"pairs": 1101,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "nb.6160fde1baeb.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "6160fde1baebbc57cd9599d56df7c1aa33b665933fa344cc3834db2f9c581c1f",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"nl": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-NL by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 196112,
"counts": {
"alphabet": 41,
"letter_contexts": 756,
"pair_contexts": 4413,
"pairs": 17920,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "nl.884afd31eb38.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "884afd31eb383ed9512a7e23157201826221c736ead1abe6135913435eab41ca",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"pa": {
"attribution": "Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.",
"bytes": 160276,
"counts": {
"alphabet": 59,
"letter_contexts": 1214,
"pair_contexts": 2531,
"pairs": 5800,
"starts": 64,
"words": 10495
},
"coverage_note": "Word frequencies from one small source: 104,833 words of sentences written for people to read aloud (Common Voice).",
"file": "pa.4960d7315d51.bin",
"licence": "CC0-1.0",
"licence_url": "https://creativecommons.org/publicdomain/zero/1.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "single_source",
"sha256": "4960d7315d51f90b55335250dc00bce2ad250ecf1fc8e7d6b852bfc2a529d31d",
"sources": [
"Mozilla Common Voice sentence corpus"
]
},
"pl": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 192370,
"counts": {
"alphabet": 41,
"letter_contexts": 830,
"pair_contexts": 4830,
"pairs": 17669,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "pl.a0300d09cfa4.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "a0300d09cfa4d6252f65aacd17316a5756504790e5518cd5144e67a63b68df16",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"pt": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 165937,
"counts": {
"alphabet": 42,
"letter_contexts": 870,
"pair_contexts": 3937,
"pairs": 10912,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "pt.670b42222570.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "670b422225707387a98066b057ed1917ae4619365119acef050f5292f78a5ab3",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"ro": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 134915,
"counts": {
"alphabet": 36,
"letter_contexts": 674,
"pair_contexts": 1561,
"pairs": 4484,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "ro.caf35531616d.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "caf35531616d2094a00e8fd36eb9dcb17f572dd65590dca9f663bf8037bdaccb",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"ru": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 150585,
"counts": {
"alphabet": 33,
"letter_contexts": 745,
"pair_contexts": 3440,
"pairs": 10512,
"starts": 64,
"words": 10838
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "ru.939d65b69516.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "939d65b695166184a0568584659c0e10427336678b2f455171968b558db94727",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"sv": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 150173,
"counts": {
"alphabet": 34,
"letter_contexts": 736,
"pair_contexts": 2450,
"pairs": 7049,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "sv.74455f043e96.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "74455f043e96ec2edd99dd82db531a71a64c8eac799ab5faf7f30bcf86818f87",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"sw": {
"attribution": "Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication.",
"bytes": 207820,
"counts": {
"alphabet": 30,
"letter_contexts": 540,
"pair_contexts": 2755,
"pairs": 19739,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from one small source: 620,853 words of sentences written for people to read aloud (Common Voice).",
"file": "sw.5213f3efd357.bin",
"licence": "CC0-1.0",
"licence_url": "https://creativecommons.org/publicdomain/zero/1.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "single_source",
"sha256": "5213f3efd357b72201419f8a7f639300493102d172822ce8f4159a4e36fdb3b4",
"sources": [
"Mozilla Common Voice sentence corpus"
]
},
"ta": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikisource and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 127576,
"counts": {
"alphabet": 46,
"letter_contexts": 907,
"pair_contexts": 1219,
"pairs": 2870,
"starts": 64,
"words": 7607
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "ta.513b79f9779a.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "513b79f9779a2d7201db174f8d5671061fdccf168ead2b30108c0398d20c4591",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"te": {
"attribution": "Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 135606,
"counts": {
"alphabet": 61,
"letter_contexts": 1361,
"pair_contexts": 0,
"pairs": 0,
"starts": 58,
"words": 7600
},
"coverage_note": "Word frequencies from one source: 112,894 words of film and TV subtitles (OpenSubtitles 2018).",
"file": "te.86a14c9608a4.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word list from film and TV subtitles (OPUS OpenSubtitles 2018), which is published with a request for attribution to OpenSubtitles rather than an explicit licence; labelled CC BY-SA 4.0 as FrequencyWords' content is",
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation",
"few word pairs (0): next-word suggestions come mostly from word frequency"
],
"quality": "single_source",
"sha256": "86a14c9608a4b3b4ef58a4b82d78dc4100ff561fcbd36e0ccb16ff7e7fde3d8a",
"sources": [
"FrequencyWords 2018 (Hermit Dave)",
"Mozilla Common Voice sentence corpus"
]
},
"th": {
"attribution": "Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 166814,
"counts": {
"alphabet": 65,
"letter_contexts": 1682,
"pair_contexts": 1499,
"pairs": 8757,
"starts": 64,
"words": 7260
},
"coverage_note": "Word frequencies from one source: 2,548,322 words of film and TV subtitles (OpenSubtitles 2018).",
"file": "th.5c77a14224c2.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word list from film and TV subtitles (OPUS OpenSubtitles 2018), which is published with a request for attribution to OpenSubtitles rather than an explicit licence; labelled CC BY-SA 4.0 as FrequencyWords' content is",
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation",
"written without spaces: word pairs were counted after dictionary (maximum-match) segmentation, which is approximate"
],
"quality": "single_source",
"sha256": "5c77a14224c243c4144986f4eb9532661c05eaeda82a18e33504be97954aa3bd",
"sources": [
"FrequencyWords 2018 (Hermit Dave)",
"Mozilla Common Voice sentence corpus"
]
},
"tl": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 126767,
"counts": {
"alphabet": 35,
"letter_contexts": 639,
"pair_contexts": 241,
"pairs": 419,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "tl.6c16deeb4dfb.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation",
"few word pairs (419): next-word suggestions come mostly from word frequency"
],
"quality": "multi_source",
"sha256": "6c16deeb4dfb8966a539e67618a2c5aec4e074549f7e31d34348f9da47db92b1",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"tr": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 187417,
"counts": {
"alphabet": 36,
"letter_contexts": 828,
"pair_contexts": 5603,
"pairs": 16562,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "tr.0ca88ed91234.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "0ca88ed91234ef2070b544423cfec3106acfb98962f9617f39e4f853fcd6ad1d",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"uk": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 149726,
"counts": {
"alphabet": 38,
"letter_contexts": 788,
"pair_contexts": 2704,
"pairs": 8996,
"starts": 64,
"words": 11018
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "uk.ee9bb286ed0e.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "ee9bb286ed0ea1d91fd46daf1e5cac8f6572a2bd2abac260be58b1502431b696",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"ur": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 173630,
"counts": {
"alphabet": 53,
"letter_contexts": 1233,
"pair_contexts": 3459,
"pairs": 9407,
"starts": 64,
"words": 14682
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "ur.ebc094140ee3.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "ebc094140ee3f2fd48bc701c0ad54b33681243c3e0e79d095e11922a683fb9d4",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"vi": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 105056,
"counts": {
"alphabet": 98,
"letter_contexts": 1971,
"pair_contexts": 1037,
"pairs": 3479,
"starts": 64,
"words": 10425
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "vi.3a419f260adb.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "multi_source",
"sha256": "3a419f260adba18a25dc7834530f7d9261034cca5f5eaa2664929af2d79e81a0",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"yo": {
"attribution": "Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.",
"bytes": 89773,
"counts": {
"alphabet": 42,
"letter_contexts": 960,
"pair_contexts": 1381,
"pairs": 4191,
"starts": 64,
"words": 5883
},
"coverage_note": "Word frequencies from one small source: 57,844 words of sentences written for people to read aloud (Common Voice). Few words: expect rough suggestions until it learns this person's words.",
"file": "yo.bdbadbccfe5a.bin",
"licence": "CC0-1.0",
"licence_url": "https://creativecommons.org/publicdomain/zero/1.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation"
],
"quality": "sparse",
"sha256": "bdbadbccfe5a6ca04ba6888e7235bee92e841f2d08214d46d7f5597224aa3967",
"sources": [
"Mozilla Common Voice sentence corpus"
]
},
"zh": {
"attribution": "Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-CH by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; the word list that comes with the Jieba segmenter, (c) 2013 Sun Junyi, MIT (https://github.com/fxsjy/jieba), whose provenance wordfreq says it does not know. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0.",
"bytes": 223934,
"counts": {
"alphabet": 400,
"letter_contexts": 1611,
"pair_contexts": 3576,
"pairs": 18922,
"starts": 64,
"words": 16000
},
"coverage_note": "Word frequencies from wordfreq, which combines at least three kinds of text (e.g. Wikipedia, subtitles, web, social media) up to about 2021.",
"file": "zh.18ae613de569.bin",
"licence": "CC-BY-SA-4.0",
"licence_url": "https://creativecommons.org/licenses/by-sa/4.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation",
"written without spaces: word pairs were counted after dictionary (maximum-match) segmentation, which is approximate"
],
"quality": "multi_source",
"sha256": "18ae613de569790276e30283c197a2ebf45bdfd3b8ef3f050ceefcdcc2a9c7c1",
"sources": [
"Mozilla Common Voice sentence corpus",
"wordfreq 3.2.0 (Robyn Speer)"
]
},
"zu": {
"attribution": "Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.",
"bytes": 44927,
"counts": {
"alphabet": 26,
"letter_contexts": 330,
"pair_contexts": 185,
"pairs": 256,
"starts": 64,
"words": 4318
},
"coverage_note": "Word frequencies from one small source: 8,394 words of sentences written for people to read aloud (Common Voice). Few words: expect rough suggestions until it learns this person's words.",
"file": "zu.8ebf1bfd8c49.bin",
"licence": "CC0-1.0",
"licence_url": "https://creativecommons.org/publicdomain/zero/1.0/",
"notes": [
"word pairs and sentence starts come from sentences written for reading aloud (much of it news and encyclopedia text), not from conversation",
"few word pairs (256): next-word suggestions come mostly from word frequency"
],
"quality": "sparse",
"sha256": "8ebf1bfd8c4932135d5928ac25e3fe6fe5a7accd877cd904f6e1d392f42ab7d1",
"sources": [
"Mozilla Common Voice sentence corpus"
]
}
},
"unicode": "16.0.0",
"version": 1
}
Reference in a new issue View git blame Copy permalink