88317db58e
H-A: by_form from a board word's fewest-endings reading, one word only; by_readings at the fewest endings among the list's words, nowhere when that word is on no board (つけて never つかれた, おくれた never おくる). H-B: a board word is a noun only where its entry in its sense is one (finder_ja.json classes "n"); the verb after を/が, or at the end after a question word or adverb; the noun alone, after の, before a case particle. M1: equal cost and one-kana words: more particles, then earlier. M2: a board's <noun>する is a する verb where the list has the noun as one. M3: two words of the list at the fewest endings lead nowhere but where ja_inflect::DOMINANT settles it; after を, not an intransitive verb (kana list "vi"; "adv" for adverbs). Co-Authored-By: Claude Opus 5.5 <[email protected]>
761 lines
40 KiB
Python
761 lines
40 KiB
Python
#!/usr/bin/env python3
|
||
"""Japanese spellings of the app's Japanese words, in the sense each is used,
|
||
for finding them (aac-core finder.rs `ja_spellings`).
|
||
|
||
A Japanese word is written in kana or in kanji, and the app's own Japanese
|
||
words are mostly kana (みず) while an input method, or the partner listener,
|
||
gives kanji (水). But a kana word is many words: the app's はな is "nose"
|
||
(鼻), never "flower" (花). So the spellings are chosen by SENSE, not by
|
||
reading.
|
||
|
||
The app's words. Every (English key, Japanese word) pair the app has: the
|
||
glosses (aac-core translations_data.json, English key -> ja), the navigation
|
||
words (nav_labels.json: "back", "home") and the phrase sets' single words
|
||
(tools/pagesets/phrases/translations/ja.json: English source -> text; a
|
||
single word is short, with no space or punctuation). The English key is the
|
||
sense: the app's glosses were audited in the sense of vocabulary.rs
|
||
(gloss_audit_2026-10-01.json), and its other languages' glosses pin it down.
|
||
|
||
`senses` — the sense rule. For each pair, the other spellings of the word in
|
||
the key's sense, from the JMdict entries whose senses mean the key:
|
||
|
||
- the candidates are the JMdict entries of the word: for a word in kana, the
|
||
entries read so whose common kanji spellings (a priority tag: news1, ichi1,
|
||
spec1, spec2, gai1; re_restr respected) are not empty; for a word with
|
||
kanji, the entries spelled so — only those with a common reading of that
|
||
spelling, when any has one (頭: not the archaic こうべ or つむり);
|
||
- an entry matches when one of its senses has a gloss equal to the key, both
|
||
normalised: lower-case, parenthesised parts dropped, a leading "to "
|
||
dropped ("cut" = "to cut"); the key is also tried without a leading "a",
|
||
"an", "the", "my", "your" (the phrase sets' "my nose") and as the singular
|
||
of a plural ("Feelings": "feeling"). A sense restricted to some spellings
|
||
(stagk) or readings (stagr) applies only to them;
|
||
- when more than one entry matches, those whose matching sense's part of
|
||
speech fits the app's word class (vocabulary.rs, lexicon.en.yaml) are
|
||
preferred — not required: the app's verb "hurt" is glossed with the
|
||
adjective いたい, which is right;
|
||
- the spellings: for a word in kana, the matching entries' common kanji
|
||
spellings that the matching sense applies to; for a word with kanji, the
|
||
matching entries' readings (not re_nokanji; re_restr and stagr respected;
|
||
not out-dated, search-only or rare kana: ok, sk, rk), the common ones only
|
||
when the entry marks any reading common.
|
||
|
||
The honorific rule. A word made of お, ご or 御 and another word (おはな,
|
||
お茶, ご家族) is the same word as that other word, its stem, in the same
|
||
sense. So when the stem has an entry in the key's sense (by the same rule,
|
||
the same review), the pair also gets the stem's spellings: a kana stem's
|
||
kanji (おはな: 花), or a kanji stem as written and its readings (お茶: 茶,
|
||
ちゃ). And the honorific's own kanji spelling counts as common when the stem
|
||
part of it is a common spelling in the sense (お花: 花), as JMdict seldom
|
||
marks the honorific forms common. A stem that has no entry in the sense
|
||
gives nothing (おなか is not なか, "inside").
|
||
|
||
The REVIEWED table. A pair whose word JMdict has candidates for but that the
|
||
automatic rule matched to none, or to more than one distinct entry (after the
|
||
part-of-speech preference) — of the word, or of its stem — is decided by
|
||
hand below, by reading the entries' senses against the key, the app's other
|
||
languages' glosses and the sense audit: `(key, ja): ([spellings], reason)`.
|
||
A decision lists only spellings the candidates give, and `[]` when none of
|
||
them is the key's sense. It may also narrow an automatic match whose JMdict
|
||
sense covers a spelling JMdict notes is another sense's (あし "foot" is 足;
|
||
脚 is esp. the leg). A pair JMdict has no candidates for gets `[]`, so the
|
||
runtime gives it no other spelling. The build stops on a pair that needs a
|
||
decision and has none, and on a decided spelling no candidate gives.
|
||
|
||
The する rule. A verb made of a noun and する (うんてんする, 料理する) is
|
||
written as its noun is, with する: its other spellings are its noun's (運転
|
||
する, りょうりする). JMdict lists the noun (with the suru-verb part of
|
||
speech), not the verb, and glosses the noun ("driving" for "drive"), so
|
||
which noun is meant is decided by hand in SURU below — `(key, ja): (noun's
|
||
other spelling, reason)` — and checked: the noun must be a JMdict entry with
|
||
that reading and spelling, marked as a suru verb. A verb with more words
|
||
before する (ゲームをする) is several words, and gets none.
|
||
|
||
`classes` — the inflecting classes of each app word in its sense, so a
|
||
verb typed in another form leads only to the board word that is that verb
|
||
(aac-web finder.rs): かえりたい is 帰る's, never the app's かえる "change" (変える),
|
||
and いらない is 要る's, never いる "stay" (居る). The word's entry in the sense is
|
||
found by its kanji where `senses` gives them (かえる "change": the entries
|
||
spelled 変える, 替える or 換える; いたい "hurt": 痛い), else by the sense rule
|
||
over all the JMdict entries of the word, read or spelled so (いる "stay": 居る,
|
||
whose kanji is not common) — when more than one matches, those whose
|
||
matching sense's part of speech fits the app's word class are preferred.
|
||
The classes are those of the entry's parts of speech (by kanji) or of its
|
||
matching senses' (by the sense rule): v1, v5, vk, vs, vsn, adj, exp, adv as
|
||
in build_ja_kana.py, and n for a noun (した "down" is 下, "n"; すみません
|
||
"excuse me" is "exp", a set phrase; よく "well" is "adv"). A pair whose entries differ in their classes, or with no entry found,
|
||
is left out: its class is not known, and no other form leads there.
|
||
|
||
`readings` — the old reading-only table, used only for a person's own words
|
||
that have no English key, ranked low and never for a message's word: for a
|
||
word in kana, the common kanji of every common JMdict entry read so; for a
|
||
word with kanji, the readings of every entry spelled so; at most three each.
|
||
|
||
Usage:
|
||
python3 tools/finder/build_ja_readings.py JMdict_e.gz
|
||
|
||
JMdict is the property of the Electronic Dictionary Research and Development
|
||
Group (EDRDG), used under its licence (CC BY-SA 4.0,
|
||
https://www.edrdg.org/edrdg/licence.html); the table this writes is shared
|
||
under the same licence (crates/aac-core/src/finder_ja.LICENSE.md).
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import gzip
|
||
import json
|
||
import re
|
||
import sys
|
||
import xml.etree.ElementTree as ET
|
||
from pathlib import Path
|
||
|
||
ROOT = Path(__file__).resolve().parents[2]
|
||
CORE = ROOT / "crates" / "aac-core" / "src"
|
||
OUT = CORE / "finder_ja.json"
|
||
PRIORITY = {"news1", "ichi1", "spec1", "spec2", "gai1"}
|
||
KANJI = re.compile(r"[㐀-䶿一-鿿豈-]")
|
||
KANA_ONLY = re.compile(r"^[-ヿー]+$")
|
||
MAX_FORMS = 3
|
||
HONORIFIC = ("お", "ご", "御")
|
||
# Kana readings left out of a kanji word's spellings: out-dated, search-only
|
||
# or rarely used kana (JMdict re_inf ok, sk, rk, as the parser expands them).
|
||
SKIP_READING = {"out-dated or obsolete kana usage", "search-only kana form", "rarely used kana form"}
|
||
# Kanji spellings never taken for an honorific's own (ke_inf sK, rK).
|
||
SKIP_KANJI = {"search-only kanji form", "rarely used kanji form"}
|
||
# A leading word of a phrase-set key that does not change its sense.
|
||
DETERMINER = re.compile(r"^(?:a|an|the|my|your) (.+)$")
|
||
|
||
# ── Reviewed decisions ──────────────────────────────────────────────────────
|
||
# (English key, ja word): ([other spellings in the key's sense], reason).
|
||
REVIEWED: dict[tuple[str, str], tuple[list[str], str]] = {
|
||
# Automatic matches narrowed: JMdict's sense covers several spellings,
|
||
# but notes that one of them is the spelling of another sense.
|
||
("foot", "あし"): (["足"], "foot, esp. 足; JMdict gives 脚 esp. for the leg (the phrase sets' 'my leg')"),
|
||
("fast", "はやい"): (["速い"], "fast (es rápido), esp. 速い; 早い is early, and fast only as quick and easy"),
|
||
("head", "頭"): (["あたま"], "head: あたま; 頭 read かしら is a chief, or the head in literary use"),
|
||
("my head", "頭"): (["あたま"], "head: あたま; 頭 read かしら is a chief, or the head in literary use"),
|
||
# Kanji words whose one entry means the key in other words.
|
||
("A person I know", "知り合い"): (["しりあい"], "acquaintance"),
|
||
("Back", "戻る"): (["もどる"], "to return; to go back"),
|
||
("back", "戻る"): (["もどる"], "to return; to go back (the navigation word)"),
|
||
("Brain injury", "脳損傷"): (["のうそんしょう"], "brain damage"),
|
||
("Brother", "兄弟"): (["きょうだい"], "brothers and sisters; brothers"),
|
||
("Care needs", "介助"): (["かいじょ"], "help; assistance (in care)"),
|
||
("Clear", "消去"): (["しょうきょ"], "erasure; clearing (fr effacer, de löschen)"),
|
||
("Core", "基本語"): (["きほんご"], "basic vocabulary item: core words"),
|
||
("Going out", "おでかけ"): (["お出かけ"], "outing; trip"),
|
||
("I feel sick", "具合が悪い"): (["ぐあいがわるい"], "unwell"),
|
||
("I'd like", "お願いします"): (["おねがいします"], "please (a request)"),
|
||
("I'm finished", "終わり"): (["おわり"], "end"),
|
||
("Introducing myself", "自己紹介"): (["じこしょうかい"], "self-introduction"),
|
||
("It hurts", "痛い"): (["いたい"], "painful; sore"),
|
||
("Let's", "一緒に"): (["いっしょに"], "together: the app's let's-do-it-together word"),
|
||
("Make a choice", "選ぶ"): (["えらぶ"], "to choose"),
|
||
("Medical", "医療"): (["いりょう"], "medical care"),
|
||
("Parkinson's", "パーキンソン病"): (["パーキンソンびょう"], "Parkinson's disease"),
|
||
("Prepositions", "助詞"): (["じょし"], "particle: Japanese's counterpart of prepositions"),
|
||
("Recent", "最近"): (["さいきん"], "recently"),
|
||
("Responses", "返答"): (["へんとう"], "reply; answer"),
|
||
("Sister", "姉妹"): (["しまい"], "sisters"),
|
||
("Starting a phone call", "電話をかける"): (["でんわをかける"], "to make a phone call"),
|
||
# Kana words whose entry means the key in other words.
|
||
("Describing", "ようす"): (["様子"], "appearance; state: how things are, the describing words' label (es describir)"),
|
||
("Social", "やりとり"): (["やり取り"], "(conversational) exchange"),
|
||
("Tell a riddle", "なぞなぞ"): (["謎々"], "riddle"),
|
||
("all done", "おわり"): (["終わり"], "end: finished (es terminado); not 割り, rate"),
|
||
("all over", "全身"): (["ぜんしん"], "the whole body"),
|
||
("angry", "腹が立つ"): (["はらがたつ"], "to get angry"),
|
||
("bored", "退屈"): (["たいくつ"], "boring; to get bored"),
|
||
("draw", "絵をかく"): (["えをかく"], "to draw a picture"),
|
||
("great", "最高"): (["さいこう"], "best; wonderful"),
|
||
("around", "まわり"): (["周り"], "surroundings; vicinity (es alrededor); not 回り, rotation"),
|
||
("ask", "きく"): (["聞く", "訊く"], "聞く 'to ask' and 訊く 'to ask'; not 聴く (listen; ask only in legal use), 効く/利く (be effective), 菊 (chrysanthemum)"),
|
||
("away", "とおく"): (["遠く"], "far away (es lejos)"),
|
||
("beach", "うみ"): (["海"], "the app's word for the beach is うみ, the sea(side): 海; not 生み, birth"),
|
||
("bored", "つまらない"): (["詰らない"], "boring"),
|
||
("can", "できる"): (["出来る"], "to be able to do"),
|
||
("change", "かえる"): (["変える", "替える", "換える"], "to change (es cambiar, zh 换): 変える and 替える/換える; not 代える (substitute a person), 帰る/返る (return), 蛙 (frog)"),
|
||
("chicken", "とりにく"): (["鶏肉"], "chicken meat"),
|
||
("cousin", "いとこ"): (["従兄弟", "従姉妹"], "cousin, male and female: the English is either"),
|
||
("curious", "きょうみ"): (["興味"], "interest; curiosity: the app's word for curious"),
|
||
("cut", "きる"): (["切る"], "to cut; not 斬る (to kill or slash with a blade), 着る (to wear)"),
|
||
("dangerous", "きけん"): (["危険"], "danger; not 棄権, abstention"),
|
||
("different", "ちがう"): (["違う"], "to be different"),
|
||
("dizzy", "めまい"): (["眩暈", "目眩"], "dizziness"),
|
||
("dog", "いぬ"): (["犬"], "dog; not 戌, the zodiac sign"),
|
||
("draw", "えをかく"): (["絵を描く"], "to draw a picture"),
|
||
("dress", "きがえる"): (["着替える"], "to change one's clothes (es vestirse)"),
|
||
("few", "すこし"): (["少し"], "a few; a little"),
|
||
("frustrated", "いらいら"): (["苛々"], "to get irritated"),
|
||
("get", "もらう"): (["貰う"], "to receive"),
|
||
("grandma", "おばあさん"): (["お祖母さん"], "grandmother; not お婆さん or 婆さん, which JMdict gives for an old woman"),
|
||
("grandpa", "おじいさん"): (["お祖父さん"], "grandfather; not お爺さん or 爺さん, which JMdict gives for an old man"),
|
||
("grateful", "かんしゃ"): (["感謝"], "gratitude"),
|
||
("hang", "かける"): (["掛ける"], "to hang up; not 駆ける (run), 欠ける (chip), 賭ける (bet), 翔る (soar)"),
|
||
("hard", "かたい"): (["硬い", "固い", "堅い"], "hard to the touch (es duro, zh 硬, ko 단단한); not 難い, difficult"),
|
||
("hate", "だいきらい"): (["大嫌い"], "hating"),
|
||
("hear", "きこえる"): (["聞こえる"], "to be heard"),
|
||
("hi", "こんにちは"): (["今日は"], "hello"),
|
||
("hold", "もつ"): (["持つ"], "to hold in one's hand (es sostener, zh 拿着); not 保つ, to keep or last"),
|
||
("hope", "のぞむ"): (["望む"], "to hope for; not 臨む, to face"),
|
||
("hot", "あつい"): (["暑い", "熱い"], "hot weather and hot to the touch (es caliente, ko 더운); not 厚い, thick"),
|
||
("house", "家"): (["いえ", "うち"], "house: 家 read いえ and うち; not け, a suffix after a surname"),
|
||
("hurt", "いたい"): (["痛い"], "painful: it hurts (es doler); not 遺体, a corpse"),
|
||
("ouch", "いたい"): (["痛い"], "painful: ouch (es ay); not 遺体, a corpse"),
|
||
("hurting", "痛い"): (["いたい"], "painful"),
|
||
("jealous", "やきもち"): (["焼きもち"], "jealousy"),
|
||
("like", "すき"): (["好き"], "liking; not 隙, a gap"),
|
||
("like", "好き"): (["すき"], "liking"),
|
||
("love", "だいすき"): (["大好き"], "loving"),
|
||
("me", "私"): (["わたし", "あたし"], "I; me: both readings of 私 in that sense"),
|
||
("my brother", "兄弟"): (["きょうだい"], "brothers and sisters; brothers"),
|
||
("my phone", "携帯"): (["けいたい"], "mobile phone"),
|
||
("my sister", "姉妹"): (["しまい"], "sisters"),
|
||
("nauseous", "はきけ"): (["吐き気"], "nausea"),
|
||
("nervous", "きんちょう"): (["緊張"], "nervousness"),
|
||
("never", "けっして"): (["決して"], "(not) ever"),
|
||
("next to", "となり"): (["隣"], "next to; adjoining"),
|
||
("nice", "やさしい"): (["優しい"], "kind; gentle (de nett, ko 착한); not 易しい, easy"),
|
||
("off", "けす"): (["消す"], "to turn off"),
|
||
("open", "開ける"): (["あける"], "to open (es abrir): 開ける read あける; not ひらける, to open out (of a view)"),
|
||
("out", "そと"): (["外"], "outside"),
|
||
("outside", "外"): (["そと"], "outside (es afuera): 外 read そと; not ほか (other) or よそ (elsewhere)"),
|
||
("rice", "ごはん"): (["ご飯", "御飯"], "cooked rice; the stem はん is no rice word"),
|
||
("ride", "のる"): (["乗る"], "to get on, ride; not 載る, to be placed on"),
|
||
("safe", "あんぜん"): (["安全"], "safety"),
|
||
("scared", "こわい"): (["怖い"], "scary; afraid"),
|
||
("sick", "びょうき"): (["病気"], "illness"),
|
||
("sick", "具合が悪い"): (["ぐあいがわるい"], "unwell"),
|
||
("sister", "おねえさん"): (["お姉さん", "姉さん"], "older sister (zh 姐姐, ko 누나), and its stem 姉さん in the same sense"),
|
||
("take", "とる"): (["取る"], "to take (es tomar, zh 拿); not 撮る (a photo), 採る (adopt, gather), 執る, 摂る (have a meal)"),
|
||
("the store", "お店"): (["おみせ", "店", "みせ"], "store: お店 and its stem 店 read みせ; not てん, 店 as a suffix"),
|
||
("tired", "疲れた"): (["つかれた"], "worn-out"),
|
||
("want", "ほしい"): (["欲しい"], "wanting"),
|
||
("where is", "どこ"): (["何処"], "where"),
|
||
("who is", "だれ"): (["誰"], "who"),
|
||
("worried", "心配"): (["しんぱい"], "worry"),
|
||
("wrong", "まちがい"): (["間違い"], "mistake"),
|
||
("yucky", "まずい"): (["不味い"], "bad-tasting"),
|
||
("yummy", "おいしい"): (["美味しい"], "delicious"),
|
||
# Kana words no candidate of which is the key's sense.
|
||
("since", "から"): ([], "the particle から; 殻 is a shell, 空 empty"),
|
||
("stay", "いる"): ([], "いる, to be or stay, has no common kanji (居る); 射る shoot, 炒る roast, 入る enter, 要る need, 鋳る cast"),
|
||
("that", "あれ"): ([], "あれ, that, has no common kanji; 我 is I, 荒れ stormy weather"),
|
||
("too", "も"): ([], "the particle も; 喪 is mourning, 藻 algae, 面 face"),
|
||
("when", "いつ"): ([], "the only candidate is 五, five"),
|
||
("with", "と"): ([], "the particle と; 外 outside, 戸 door, 斗, 十 ten, 途 way"),
|
||
("won't", "しない"): ([], "not doing; 市内 is within the city, 竹刀 a bamboo sword"),
|
||
("yes", "はい"): ([], "灰 is ash, 肺 lung"),
|
||
("here", "ここ"): ([], "此処 is not common; 九 is nine, 個々 individual"),
|
||
("how", "どう"): ([], "どう, how, has no common kanji; 動 is motion, 同 same, 堂 hall, 胴 trunk, 銅 copper"),
|
||
("also", "も"): ([], "the particle も; 喪 is mourning, 藻 algae, 面 face"),
|
||
("do", "する"): ([], "する, to do, has no common kanji; 刷る is print, 剃る shave, 擦る rub"),
|
||
("When", "いつ"): ([], "the only candidate is 五, five"),
|
||
("Yes", "はい"): ([], "灰 is ash, 肺 lung"),
|
||
}
|
||
|
||
|
||
# The する rule: the noun's other spelling, by the verb's sense.
|
||
SURU: dict[tuple[str, str], tuple[str, str]] = {
|
||
("drive", "うんてんする"): ("運転", "driving (a vehicle)"),
|
||
("agree", "さんせいする"): ("賛成", "approval; agreement"),
|
||
("worry", "しんぱいする"): ("心配", "worry; concern"),
|
||
("explain", "せつめいする"): ("説明", "explanation"),
|
||
("Describe it", "説明する"): ("せつめい", "explanation: describing it"),
|
||
("call", "でんわする"): ("電話", "telephone call: to phone"),
|
||
("promise", "やくそくする"): ("約束", "promise"),
|
||
("cook", "りょうりする"): ("料理", "cooking"),
|
||
("cook", "料理する"): ("りょうり", "cooking"),
|
||
}
|
||
|
||
|
||
# ── The app's words ─────────────────────────────────────────────────────────
|
||
|
||
|
||
def single_word(text: str) -> bool:
|
||
"""A phrase-set text that is one word: no space, no punctuation, short."""
|
||
return bool(text) and len(text) <= 8 and not re.search(r"[\s、。！？!?,.・「」()（）0-9]", text)
|
||
|
||
|
||
def is_japanese(word: str) -> bool:
|
||
return bool(word) and bool(KANJI.search(word) or KANA_ONLY.match(word))
|
||
|
||
|
||
def app_pairs() -> set[tuple[str, str]]:
|
||
"""Every (English key, ja word) pair the app has."""
|
||
pairs: set[tuple[str, str]] = set()
|
||
data = json.loads((CORE / "translations_data.json").read_text(encoding="utf-8"))
|
||
for key, value in data.items():
|
||
if isinstance(value, dict) and isinstance(value.get("ja"), str):
|
||
pairs.add((key, value["ja"].strip()))
|
||
nav = json.loads((CORE / "nav_labels.json").read_text(encoding="utf-8"))
|
||
ja = nav.get("languages", {}).get("ja", {})
|
||
for key in ("back", "home"):
|
||
if isinstance(ja.get(key), str):
|
||
pairs.add((key, ja[key].strip()))
|
||
phrases = ROOT / "tools" / "pagesets" / "phrases" / "translations" / "ja.json"
|
||
if phrases.exists():
|
||
for key, entry in json.loads(phrases.read_text(encoding="utf-8")).get("strings", {}).items():
|
||
text = (entry or {}).get("text", "").strip()
|
||
if single_word(text):
|
||
pairs.add((key, text))
|
||
return {(k, w) for k, w in pairs if is_japanese(w)}
|
||
|
||
|
||
def word_classes() -> dict[str, str]:
|
||
"""English word -> the app's word class (vocabulary.rs, then lexicon.en.yaml)."""
|
||
classes: dict[str, str] = {}
|
||
lexicon = ROOT / "tools" / "pagesets" / "data" / "lexicon.en.yaml"
|
||
if lexicon.exists():
|
||
for line in lexicon.read_text(encoding="utf-8").splitlines():
|
||
m = re.match(r"^\s+(['\"]?)(.+?)\1:\s*\{.*\bpos:\s*([a-z_]+)", line)
|
||
if m:
|
||
classes[m.group(2).lower()] = m.group(3)
|
||
vocabulary = (CORE / "vocabulary.rs").read_text(encoding="utf-8")
|
||
for word, category in re.findall(r'\("([^"]+)", WordCategory::(\w+)', vocabulary):
|
||
classes[word.lower()] = category.lower()
|
||
return classes
|
||
|
||
|
||
# ── Matching ────────────────────────────────────────────────────────────────
|
||
|
||
|
||
def norm(text: str) -> str:
|
||
text = re.sub(r"\([^)]*\)", " ", text.lower().replace("’", "'"))
|
||
text = " ".join(text.split())
|
||
return text[3:] if text.startswith("to ") else text
|
||
|
||
|
||
def key_forms(key: str) -> set[str]:
|
||
"""The key as glosses spell it: also without a leading determiner ("my
|
||
nose"), and a plural as its singular ("Feelings", "eggs", "Stories")."""
|
||
k = norm(key)
|
||
forms = {k}
|
||
m = DETERMINER.match(k)
|
||
if m:
|
||
forms.add(norm(m.group(1)))
|
||
for form in list(forms):
|
||
if form.endswith("ies") and len(form) > 4:
|
||
forms.add(form[:-3] + "y")
|
||
elif form.endswith(("ses", "xes", "zes", "ches", "shes")):
|
||
forms.add(form[:-2])
|
||
elif form.endswith("s") and not form.endswith("ss") and len(form) > 3:
|
||
forms.add(form[:-1])
|
||
return forms - {""}
|
||
|
||
|
||
def pos_fits(app_class: str | None, pos: list[str]) -> bool:
|
||
if not app_class:
|
||
return False
|
||
text = " ".join(pos).lower()
|
||
if app_class == "verb":
|
||
return ("verb" in text and "noun" not in text) or "suru verb" in text
|
||
if app_class in ("corenoun", "fringenoun", "core_noun", "fringe_noun", "noun"):
|
||
return "noun" in text
|
||
if app_class in ("adjective", "adverb", "negation"):
|
||
return "adjective" in text or "adjectival" in text or "adverb" in text
|
||
if app_class == "pronoun":
|
||
return "pronoun" in text
|
||
if app_class in ("social", "interjection", "phrase"):
|
||
return "interjection" in text or "expression" in text
|
||
if app_class in ("preposition", "determiner", "question", "conjunction"):
|
||
return "noun" in text or "pronoun" in text or "adverb" in text
|
||
return False
|
||
|
||
|
||
class Entry:
|
||
__slots__ = ("seq", "kebs", "rebs", "senses")
|
||
|
||
def __init__(self, el: ET.Element):
|
||
self.seq = el.findtext("ent_seq")
|
||
# (spelling, priority tags, ke_inf)
|
||
self.kebs = [
|
||
(k.findtext("keb"), {p.text for p in k.findall("ke_pri")}, {i.text for i in k.findall("ke_inf")})
|
||
for k in el.findall("k_ele")
|
||
]
|
||
# (reading, priority tags, re_nokanji, re_restr, re_inf)
|
||
self.rebs = [
|
||
(
|
||
r.findtext("reb"),
|
||
{p.text for p in r.findall("re_pri")},
|
||
r.find("re_nokanji") is not None,
|
||
[x.text for x in r.findall("re_restr")],
|
||
{i.text for i in r.findall("re_inf")},
|
||
)
|
||
for r in el.findall("r_ele")
|
||
]
|
||
# (glosses, pos, stagk, stagr); a sense with no pos takes the last one's.
|
||
self.senses = []
|
||
pos: list[str] = []
|
||
for s in el.findall("sense"):
|
||
pos = [p.text for p in s.findall("pos")] or pos
|
||
self.senses.append(
|
||
(
|
||
[g.text or "" for g in s.findall("gloss")],
|
||
pos,
|
||
[x.text for x in s.findall("stagk")],
|
||
[x.text for x in s.findall("stagr")],
|
||
)
|
||
)
|
||
|
||
def kanji_of(self, reading: str, extra: set[str] = frozenset()) -> list[str]:
|
||
"""The kanji spellings of a reading: common ones (or in `extra`), re_restr respected."""
|
||
out: list[str] = []
|
||
for reb, _, nokanji, restr, _ in self.rebs:
|
||
if reb != reading or nokanji:
|
||
continue
|
||
for keb, pri, inf in self.kebs:
|
||
if (pri & PRIORITY or (keb in extra and not inf & SKIP_KANJI)) and (not restr or keb in restr) and keb not in out:
|
||
out.append(keb)
|
||
return out
|
||
|
||
def reads_commonly(self, spelling: str) -> bool:
|
||
"""Whether a common reading of the entry applies to this spelling."""
|
||
return any(
|
||
pri & PRIORITY and not nokanji and (not restr or spelling in restr)
|
||
for _, pri, nokanji, restr, _ in self.rebs
|
||
)
|
||
|
||
def readings_of(self, spelling: str) -> list[str]:
|
||
"""The readings of a kanji spelling: not re_nokanji, re_restr respected, common first."""
|
||
found = [
|
||
(reb, bool(pri & PRIORITY))
|
||
for reb, pri, nokanji, restr, inf in self.rebs
|
||
if not nokanji and (not restr or spelling in restr) and not inf & SKIP_READING
|
||
]
|
||
if any(common for _, common in found):
|
||
found = [x for x in found if x[1]]
|
||
return [reb for reb, _ in found]
|
||
|
||
|
||
class Dictionary:
|
||
def __init__(self, jmdict: str, words: set[str]):
|
||
self.by_reading: dict[str, list[Entry]] = {}
|
||
self.by_spelling: dict[str, list[Entry]] = {}
|
||
with gzip.open(jmdict, "rb") as f:
|
||
for _, el in ET.iterparse(f, events=("end",)):
|
||
if el.tag != "entry":
|
||
continue
|
||
kebs = [k.findtext("keb") for k in el.findall("k_ele")]
|
||
rebs = [r.findtext("reb") for r in el.findall("r_ele")]
|
||
if words.intersection(kebs) or words.intersection(rebs):
|
||
entry = Entry(el)
|
||
for keb in set(kebs) & words:
|
||
self.by_spelling.setdefault(keb, []).append(entry)
|
||
for reb in set(rebs) & words:
|
||
self.by_reading.setdefault(reb, []).append(entry)
|
||
el.clear()
|
||
|
||
def candidates(self, word: str, extra: set[str] = frozenset()) -> list[Entry]:
|
||
if KANA_ONLY.match(word):
|
||
return [e for e in self.by_reading.get(word, []) if e.kanji_of(word, extra)]
|
||
entries = self.by_spelling.get(word, [])
|
||
common = [e for e in entries if e.reads_commonly(word)]
|
||
return common or list(entries)
|
||
|
||
def sense_spellings(self, entry: Entry, word: str, forms: set[str], extra: set[str] = frozenset()):
|
||
"""The spellings `entry` gives `word` in a sense glossed by `forms`,
|
||
and the part of speech of the matching senses; None when no sense
|
||
matches."""
|
||
kana = bool(KANA_ONLY.match(word))
|
||
spellings: list[str] = []
|
||
pos: list[str] = []
|
||
matched = False
|
||
for glosses, sense_pos, stagk, stagr in entry.senses:
|
||
if not any(norm(g) in forms for g in glosses):
|
||
continue
|
||
if kana:
|
||
if stagr and word not in stagr:
|
||
continue
|
||
found = [k for k in entry.kanji_of(word, extra) if not stagk or k in stagk]
|
||
else:
|
||
if stagk and word not in stagk:
|
||
continue
|
||
found = [r for r in entry.readings_of(word) if not stagr or r in stagr]
|
||
matched = True
|
||
pos += sense_pos
|
||
spellings += [s for s in found if s not in spellings]
|
||
return (spellings, pos) if matched else None
|
||
|
||
|
||
class Decision:
|
||
"""How a word was matched in a sense: the matched entries and spellings."""
|
||
|
||
def __init__(self, dictionary: Dictionary, key: str, word: str, app_class: str | None, extra: set[str] = frozenset()):
|
||
self.word, self.extra = word, extra
|
||
forms = key_forms(key)
|
||
self.candidates = dictionary.candidates(word, extra)
|
||
matched = []
|
||
for entry in self.candidates:
|
||
found = dictionary.sense_spellings(entry, word, forms, extra)
|
||
if found is not None:
|
||
matched.append((entry, found[0], found[1]))
|
||
if len(matched) > 1:
|
||
fitting = [m for m in matched if pos_fits(app_class, m[2])]
|
||
if len(fitting) == 1:
|
||
matched = fitting
|
||
self.matched = matched
|
||
self.ambiguous = len(matched) > 1
|
||
|
||
def spellings(self) -> list[str]:
|
||
out: list[str] = []
|
||
for _, found, _ in self.matched:
|
||
out += [s for s in found if s not in out]
|
||
return out
|
||
|
||
def all_spellings(self) -> list[str]:
|
||
"""Every spelling the candidates could give (to check a review)."""
|
||
out: list[str] = []
|
||
kana = bool(KANA_ONLY.match(self.word))
|
||
for entry in self.candidates:
|
||
for s in entry.kanji_of(self.word, self.extra) if kana else entry.readings_of(self.word):
|
||
if s not in out:
|
||
out.append(s)
|
||
return out
|
||
|
||
|
||
def stem_of(word: str) -> str | None:
|
||
for prefix in HONORIFIC:
|
||
if word.startswith(prefix) and len(word) > len(prefix):
|
||
return word[len(prefix):]
|
||
return None
|
||
|
||
|
||
class Pair:
|
||
"""One (English key, ja word) pair, decided automatically."""
|
||
|
||
def __init__(self, dictionary: Dictionary, key: str, word: str, classes: dict[str, str]):
|
||
self.key, self.word = key, word
|
||
forms = key_forms(key)
|
||
app_class = next((classes[f] for f in sorted(forms) if f in classes), None)
|
||
self.stem = None
|
||
stem = stem_of(word)
|
||
extra: set[str] = set()
|
||
if stem:
|
||
self.stem = Decision(dictionary, key, stem, app_class)
|
||
if self.stem.matched:
|
||
# The honorific's own kanji: the prefix and the stem's kanji.
|
||
stem_kanji = self.stem.spellings() if KANA_ONLY.match(stem) else [stem]
|
||
extra = {word[: len(word) - len(stem)] + s for s in stem_kanji}
|
||
self.own = Decision(dictionary, key, word, app_class, extra)
|
||
|
||
def automatic(self) -> list[str]:
|
||
out = self.own.spellings()
|
||
if self.stem and self.stem.matched:
|
||
stem = self.stem.word
|
||
of_stem = ([] if KANA_ONLY.match(stem) else [stem]) + self.stem.spellings()
|
||
out += [s for s in of_stem if s not in out]
|
||
return [s for s in out if s != self.word]
|
||
|
||
def possible(self) -> list[str]:
|
||
out = self.own.all_spellings()
|
||
if self.stem:
|
||
stem = self.stem.word
|
||
out += [s for s in ([] if KANA_ONLY.match(stem) else [stem]) + self.stem.all_spellings() if s not in out]
|
||
return [s for s in out if s != self.word]
|
||
|
||
def matched(self) -> bool:
|
||
return bool(self.own.matched or (self.stem and self.stem.matched))
|
||
|
||
def needs_review(self) -> str | None:
|
||
if self.own.ambiguous:
|
||
return "several entries match"
|
||
if self.stem and self.stem.ambiguous:
|
||
return "several entries of the stem match"
|
||
if self.own.candidates and not self.matched():
|
||
return "no candidate matches"
|
||
return None
|
||
|
||
|
||
# ── The reading-only table ──────────────────────────────────────────────────
|
||
|
||
|
||
def reading_table(jmdict: str, words: set[str]) -> dict[str, list[str]]:
|
||
"""The old table: every common entry's kanji by reading, every reading by
|
||
kanji, at most three each."""
|
||
kana_to_kanji: dict[str, list[str]] = {}
|
||
kanji_to_kana: dict[str, list[str]] = {}
|
||
with gzip.open(jmdict, "rb") as f:
|
||
for _, el in ET.iterparse(f, events=("end",)):
|
||
if el.tag != "entry":
|
||
continue
|
||
kebs = [(k.findtext("keb"), {p.text for p in k.findall("ke_pri")}) for k in el.findall("k_ele")]
|
||
rebs = [(r.findtext("reb"), {p.text for p in r.findall("re_pri")}, r.find("re_nokanji") is not None) for r in el.findall("r_ele")]
|
||
for reb, rpri, nokanji in rebs:
|
||
if reb in words and not nokanji and rpri & PRIORITY:
|
||
for keb, kpri in kebs:
|
||
if kpri & PRIORITY and keb not in kana_to_kanji.setdefault(reb, []):
|
||
kana_to_kanji[reb].append(keb)
|
||
for keb, _ in kebs:
|
||
if keb in words:
|
||
for reb, _, nokanji in rebs:
|
||
if not nokanji and reb not in kanji_to_kana.setdefault(keb, []):
|
||
kanji_to_kana[keb].append(reb)
|
||
el.clear()
|
||
table = {}
|
||
for word in sorted(words):
|
||
forms = kana_to_kanji.get(word, [])[:MAX_FORMS] + kanji_to_kana.get(word, [])[:MAX_FORMS]
|
||
if forms:
|
||
table[word] = forms
|
||
return table
|
||
|
||
|
||
def suru_noun(word: str) -> str | None:
|
||
"""The noun of a noun-and-する verb written as one word, if it is one."""
|
||
if word.endswith("する") and len(word) > 3 and "を" not in word:
|
||
return word[:-2]
|
||
return None
|
||
|
||
|
||
def suru_check(dictionary: Dictionary, noun: str, other: str) -> bool:
|
||
"""Whether JMdict has the noun spelled both ways, as a suru verb."""
|
||
kana, kanji = (noun, other) if KANA_ONLY.match(noun) else (other, noun)
|
||
for entry in dictionary.by_reading.get(kana, []):
|
||
spelled = any(keb == kanji for keb, _, _ in entry.kebs)
|
||
suru = any("verb suru" in " ".join(pos).lower() or "suru verb" in " ".join(pos).lower() for _, pos, _, _ in entry.senses)
|
||
if spelled and suru:
|
||
return True
|
||
return False
|
||
|
||
|
||
def sense_classes(
|
||
dictionary: Dictionary, codes: dict[str, str], key: str, word: str, app_class: str | None, spellings: list[str]
|
||
) -> str | None:
|
||
"""The inflecting classes of `word` in the sense `key` (`classes` above),
|
||
or None when they are not known."""
|
||
from build_ja_kana import ORDER, word_class
|
||
|
||
def classes_of(pos: list[str]) -> frozenset[str]:
|
||
names = [codes.get(p, "") for p in pos]
|
||
found = {c for p in names for c in [word_class(p)] if c}
|
||
# A noun (n, n-adv, n-t…): the word that a verb's form spelled the
|
||
# same is not, unless where a verb is (した "down", 下).
|
||
if any(p == "n" or p.startswith("n-") for p in names):
|
||
found.add("n")
|
||
return frozenset(found)
|
||
|
||
def one(found: set[frozenset[str]]) -> str | None:
|
||
if len(found) != 1:
|
||
return None
|
||
(classes,) = found
|
||
return ",".join(c for c in [*ORDER, "n"] if c in classes)
|
||
|
||
kana = bool(KANA_ONLY.match(word))
|
||
entries = dictionary.by_reading.get(word, []) if kana else dictionary.by_spelling.get(word, [])
|
||
# By the kanji of the sense, for a word in kana.
|
||
if kana and spellings:
|
||
by_kanji = [e for e in entries if any(keb in spellings for keb, _, _ in e.kebs)]
|
||
if by_kanji:
|
||
return one({classes_of([p for _, pos, _, _ in e.senses for p in pos]) for e in by_kanji})
|
||
forms = key_forms(key)
|
||
matched: list[list[str]] = []
|
||
for entry in entries:
|
||
pos: list[str] = []
|
||
hit = False
|
||
for glosses, sense_pos, stagk, stagr in entry.senses:
|
||
if not any(norm(g) in forms for g in glosses):
|
||
continue
|
||
if (kana and stagr and word not in stagr) or (not kana and stagk and word not in stagk):
|
||
continue
|
||
hit = True
|
||
pos += sense_pos
|
||
if hit:
|
||
matched.append(pos)
|
||
if len(matched) > 1:
|
||
fitting = [pos for pos in matched if pos_fits(app_class, pos)]
|
||
if fitting:
|
||
matched = fitting
|
||
return one({classes_of(pos) for pos in matched})
|
||
|
||
|
||
def class_table(jmdict: str, decided: dict, dictionary: Dictionary, senses: dict[str, dict[str, list[str]]]) -> dict[str, dict[str, str]]:
|
||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||
from build_ja_kana import pos_codes
|
||
|
||
codes = pos_codes(jmdict)
|
||
word_class_of = word_classes()
|
||
table: dict[str, dict[str, str]] = {}
|
||
for key, word in decided:
|
||
app_class = next((word_class_of[f] for f in sorted(key_forms(key)) if f in word_class_of), None)
|
||
found = sense_classes(dictionary, codes, key, word, app_class, senses.get(word, {}).get(key, []))
|
||
if found is not None:
|
||
table.setdefault(word, {})[key] = found
|
||
return table
|
||
|
||
|
||
def build(jmdict: str):
|
||
pairs = app_pairs()
|
||
words = {w for _, w in pairs}
|
||
stems = {s for s in map(stem_of, words) if s}
|
||
nouns = {n for n in map(suru_noun, words) if n} | {other for other, _ in SURU.values()}
|
||
dictionary = Dictionary(jmdict, words | stems | nouns)
|
||
classes = word_classes()
|
||
decided = {pair: Pair(dictionary, pair[0], pair[1], classes) for pair in sorted(pairs)}
|
||
return decided, words, dictionary
|
||
|
||
|
||
def main(jmdict: str) -> None:
|
||
decided, words, dictionary = build(jmdict)
|
||
senses: dict[str, dict[str, list[str]]] = {}
|
||
automatic = reviewed = reviewed_empty = 0
|
||
undecided, wrong = [], []
|
||
for (key, word), pair in decided.items():
|
||
if (key, word) in SURU:
|
||
other, _ = SURU[(key, word)]
|
||
noun = suru_noun(word)
|
||
if noun is None or not suru_check(dictionary, noun, other):
|
||
wrong.append(f"{key!r} {word}: {other} is not {noun} as a suru verb in JMdict")
|
||
spellings = [other + "する"]
|
||
reviewed += 1
|
||
elif (key, word) in REVIEWED:
|
||
spellings = list(REVIEWED[(key, word)][0])
|
||
stray = [s for s in spellings if s not in pair.possible()]
|
||
if stray:
|
||
wrong.append(f"{key!r} {word}: {stray} not among {pair.possible()}")
|
||
reviewed += 1
|
||
reviewed_empty += not spellings
|
||
elif pair.needs_review():
|
||
undecided.append(f"{key!r} {word}: {pair.needs_review()}")
|
||
continue
|
||
else:
|
||
spellings = pair.automatic()
|
||
automatic += pair.matched()
|
||
senses.setdefault(word, {})[key] = spellings
|
||
stale = sorted((set(REVIEWED) | set(SURU)) - set(decided))
|
||
for key, word in stale:
|
||
print(f"warning: REVIEWED has {key!r} {word}, which the app no longer has", file=sys.stderr)
|
||
if undecided or wrong:
|
||
for line in undecided:
|
||
print(f"needs a decision in REVIEWED: {line}", file=sys.stderr)
|
||
for line in wrong:
|
||
print(f"REVIEWED spelling not from JMdict: {line}", file=sys.stderr)
|
||
sys.exit(1)
|
||
OUT.write_text(
|
||
json.dumps(
|
||
{
|
||
"about": "Other spellings of the app's Japanese words, for finding them. `senses`: per Japanese word and the English key it glosses, its other spellings in that sense only (the kanji of a word the app writes in kana, or the readings of one it writes with kanji), from the JMdict entries whose senses mean the key (はな for \"nose\" is 鼻, never 花). `classes`: per Japanese word and English key, the inflecting classes of the word in that sense (v1, v5, vk, vs, vsn, adj, exp for a set phrase, adv for an adverb, n for a noun), where JMdict's entries in the sense agree (かえる \"change\" is 変える, v1). `readings`: every common spelling read or spelled the same, for words of the person's own with no English key; they may be another word. Made by tools/finder/build_ja_readings.py from JMdict (EDRDG), CC BY-SA 4.0: finder_ja.LICENSE.md.",
|
||
"source": "JMdict_e (EDRDG), https://www.edrdg.org/jmdict/j_jmdict.html",
|
||
"licence": "CC BY-SA 4.0",
|
||
"senses": senses,
|
||
"classes": class_table(jmdict, decided, dictionary, senses),
|
||
"readings": reading_table(jmdict, words),
|
||
},
|
||
ensure_ascii=False,
|
||
indent=1,
|
||
sort_keys=True,
|
||
)
|
||
+ "\n",
|
||
encoding="utf-8",
|
||
)
|
||
total = sum(len(v) for v in senses.values())
|
||
print(
|
||
f"{total} pairs ({len(senses)} Japanese words): {automatic} matched automatically, "
|
||
f"{reviewed} reviewed ({reviewed_empty} with no other spelling), "
|
||
f"{total - automatic - reviewed} with no JMdict entry in the sense -> {OUT}"
|
||
)
|
||
|
||
|
||
if __name__ == "__main__":
|
||
if len(sys.argv) != 2:
|
||
sys.exit(__doc__)
|
||
main(sys.argv[1])
|