# Language pack data — licences Generated by `aac/aac-rs/tools/lang/build_packs.py` v2; do not edit by hand. The files in this directory are **data, not code**, and are **not** covered by the Apache-2.0 licence of the reference code around them. Each `..bin` pack is licensed as stated in its own header and in `index.json`: - **Creative Commons Attribution-ShareAlike 4.0 International** (`CC-BY-SA-4.0`, https://creativecommons.org/licenses/by-sa/4.0/): `ar`, `bn`, `cs`, `da`, `de`, `el`, `en`, `es`, `fa`, `fi`, `fr`, `he`, `hi`, `hu`, `id`, `it`, `ja`, `ko`, `ml`, `ms`, `nb`, `nl`, `pl`, `pt`, `ro`, `ru`, `sv`, `ta`, `te`, `th`, `tl`, `tr`, `uk`, `ur`, `vi`, `zh`. - **CC0 1.0 Universal (public domain dedication)** (`CC0-1.0`, https://creativecommons.org/publicdomain/zero/1.0/): `am`, `ha`, `mr`, `pa`, `sw`, `yo`, `zu`. A CC BY-SA 4.0 pack is an adaptation (word, word-pair and letter statistics only) of the sources credited below, and is shared under the same licence: if you share it or an adaptation of it, keep the attribution and the licence. ShareAlike applies to the pack files, not to software that loads them. `index.json` repeats each pack's licence, attribution and coverage note; it is licensed CC BY-SA 4.0, as it carries the attributions of CC BY-SA packs. Languages without a pack (`gu`, `kn`) have no data here. ## Attribution, per pack - `am` (CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. - `ar` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `bn` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `cs` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `da` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `de` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-DE by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `el` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `en` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-US and SUBTLEX-UK by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `es` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `fa` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `fi` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `fr` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `ha` (CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. - `he` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `hi` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `hu` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `id` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `it` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `ja` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `ko` (CC-BY-SA-4.0): Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `ml` (CC-BY-SA-4.0): Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `mr` (CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. - `ms` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `nb` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `nl` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-NL by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `pa` (CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. - `pl` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Europarl and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `pt` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `ro` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `ru` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `sv` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `sw` (CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. - `ta` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikisource and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `te` (CC-BY-SA-4.0): Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `th` (CC-BY-SA-4.0): Word counts from FrequencyWords, (c) 2016 Hermit Dave (https://github.com/hermitdave/FrequencyWords), content CC BY-SA 4.0, generated from OPUS OpenSubtitles 2018, which originates from OpenSubtitles (http://www.opensubtitles.org). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `tl` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `tr` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `uk` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR; word statistics from Twitter; Reddit comments. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `ur` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); web text from OSCAR; word statistics from Twitter. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `vi` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org); web text from OSCAR. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `yo` (CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0. - `zh` (CC-BY-SA-4.0): Word frequencies from wordfreq 3.2.0, (c) 2022 Robyn Speer (https://github.com/rspeer/wordfreq, doi:10.5281/zenodo.7199437), data CC BY-SA 4.0. For this language it combines Wikipedia (https://www.wikipedia.org); subtitles from OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), which originates from OpenSubtitles (http://www.opensubtitles.org), and SUBTLEX-CH by Marc Brysbaert et al. (http://crr.ugent.be/programs-data/subtitle-frequencies; SUBTLEX is freely available data); news from NewsCrawl 2014 and GlobalVoices; Google Books Ngrams 2012 (http://books.google.com/ngrams); web text from OSCAR; word statistics from Twitter; the word list that comes with the Jieba segmenter, (c) 2013 Sun Junyi, MIT (https://github.com/fxsjy/jieba), whose provenance wordfreq says it does not know. wordfreq's licence notice also credits the Leeds Internet Corpus (University of Leeds Centre for Translation Studies) and ParaCrawl (https://paracrawl.eu). Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0, including sentences Mozilla drew from Wikipedia and releases under the same dedication. This pack is an adaptation (word, word-pair and letter statistics only) and is shared under CC BY-SA 4.0. - `zu` (CC0-1.0): Sentences from the Mozilla Common Voice sentence corpus (https://github.com/common-voice/common-voice), dedicated to the public domain under CC0 1.0.