# A text dictionary, measured against plain Zstandard > **Status.** A measured record and its procedure, informative. Every figure below is read > from [`text-dictionary/result.json`](text-dictionary/result.json), and > `research/text-dictionary/measure.py check`, which needs no Zstandard, holds the result to > the committed rule, split and dictionaries, recomputes every total from the per-object > lengths and checks every figure this page quotes (each is followed by a `txd` marker naming > its place in the result). Measured on 6 November 2026. > The record decides whether a dictionary-coded text capability is worth registering, and > §5 says it is not: no capability, vector or second implementation follows from it. ## 1. The question `wai.text.zstd` codes each object alone, with no dictionary form (SPEC §5). A Zstandard dictionary (RFC 8878 §5) lets a small object refer to text a sink already holds. That helps only if the object saves more than the dictionary costs, so the question is the one [prior-carriage](../extensions/prior-carriage.md) §17.2 asks of any set: do the bytes it saves a sink exceed what holding it costs that sink? A dictionary's bytes are counted in full for every sink that fetches it, never divided among the sinks a publisher serves. ## 2. Corpus The corpus is this repository's own text, under wai/LICENSE, copied into [`text-dictionary/corpus.tar.xz`](text-dictionary/corpus.tar.xz) with [`make_corpus.py`](text-dictionary/make_corpus.py); `corpus.json` lists each object's source, its SHA-256 and its side of the split. Four families, each object one payload a sink could fetch: - **`notes`**: the changelog's entries, one object each. - **`spec`**: the documents of `wai/extensions/`. - **`code`**: the source files of `wai/wai-rs/src/` of at most 32 KiB. - **`records`**: the JSON files of at most 8 KiB in seven conformance corpora. An object whose bytes repeat an earlier object of its family is dropped. No held-out object trains a dictionary. `notes` is split by time, the newest quarter held out, as a dictionary meets what is written after it. `spec` and `code` are split by the digest of the path. `records` is split by the digest of the directory, so that a held-out record's near-duplicates are held out with it and the dictionary meets record forms it has not seen. | family | objects, train | bytes, train | objects, held out | bytes, held out | |---|---|---|---|---| | `notes` | 234 | 868,007 | 77 | 231,828 | | `spec` | 78 | 3,145,397 | 27 | 1,571,209 | | `code` | 306 | 3,883,202 | 101 | 1,184,061 | | `records` | 636 | 916,866 | 374 | 877,014 | ## 3. The rule, and when it was fixed [`text-dictionary/rule.json`](text-dictionary/rule.json), the corpus, its split and `measure.py` were committed together before the first measurement, and the commit that holds the result follows them. The rule: - **Levels and sizes.** Levels 3, 9 and 19, and dictionaries of 4, 16, 64 and 112 KiB. Plain and dictionary forms of an object are coded at the same level, with no checksum and no dictionary id in the frame. - **Training.** One training a family and size, on the train split only, finalised per level. - **Charges.** The dictionary's whole length, once for the sink. Each object coded with it also pays 113 bytes, a minimal SPEC §3.1 `prior` in its manifest, which a plain object does not carry. An object whose dictionary form, with its pin, is not smaller than its plain form is sent plain and saves nothing (prior-carriage §17.2 step 1). - **Workload.** One sink fetches every held-out object of the family once. - **Passing.** A family, a level and a size pass when the net saving is at least 10 % of the family's plain bytes and the median object's saving is above 0. A family is GO when at every level some size passes. The families that are GO justify a capability, and when none is GO the question is closed. 10 % is the same bar as a catalogue prior's ([catalogue-prior](catalogue-prior.md) §2): a capability costs a registration, a decoder in every sink and vectors, and a saving below that does not pay for them. ## 4. Results The run read Python 3.15.2's `compression.zstd` over libzstd 1.5.7, on arm64 macOS, and a run of Python 3.15.0 over the same library on x86_64 Linux gave every dictionary and every length the same. Every dictionary form was decoded back to its object. Each row gives the dictionary size with the largest net at that level. *Total* is what the sink moves: each object in its smaller form, then the dictionary. *Net* is plain less total, as a share of plain. *Break-even* is the dictionary's bytes over the mean saving per object. *One plain payload* is the same objects coded together as one plain object, for scale; the rule does not read it. | family | level | plain bytes | dictionary | total with the dictionary | net | median saving | break-even, objects | one plain payload | |---|---|---|---|---|---|---|---|---| | notes | 3 | 103,158 | 4 KiB | 103,796 | -0.6% | 28 | 91 | 80,368 | | notes | 9 | 100,257 | 4 KiB | 98,734 | +1.5% | 63 | 56 | 73,645 | | notes | 19 | 99,138 | 4 KiB | 97,460 | +1.7% | 64 | 55 | 69,419 | | spec | 3 | 554,829 | 16 KiB | 549,190 | +1.0% | 960 | 20 | 510,830 | | spec | 9 | 515,680 | 16 KiB | 505,710 | +1.9% | 1130 | 17 | 460,909 | | spec | 19 | 492,599 | 16 KiB | 481,372 | +2.3% | 1192 | 16 | 418,576 | | code | 3 | 402,728 | 16 KiB | 388,049 | +3.6% | 303 | 53 | 328,271 | | code | 9 | 379,006 | 16 KiB | 351,549 | +7.2% | 420 | 38 | 292,188 | | code | 19 | 369,072 | 16 KiB | 339,365 | +8.0% | 450 | 36 | 266,732 | | records | 3 | 315,138 | 4 KiB | 314,108 | +0.3% | 0 | 299 | 57,762 | | records | 9 | 306,960 | 4 KiB | 305,481 | +0.5% | 0 | 275 | 52,278 | | records | 19 | 299,228 | 4 KiB | 297,638 | +0.5% | 0 | 269 | 49,968 | ## 5. Verdict against the rule No family is GO, at any level. The largest net is 8.0% (`code`, level 19, 16 KiB), then 2.3% (`spec`), 1.7% (`notes`) and 0.5% (`records`). So a dictionary-coded text capability is not registered, and this record closes the question on this corpus and this workload (§7). What the figures show: 1. **The dictionary's own bytes are the cost.** Each family's best size is its smallest or next smallest: at 64 and 112 KiB the median object still saves in every family but `records`, and the net is below zero in every family at level 3. 2. **Large objects have little to gain.** A `spec` document already holds its own repetition: its level-19 median saving at 16 KiB is 1192 bytes on a plain form of tens of kilobytes. 3. **Small objects meet the pin.** The `records` median object saves nothing: most held-out records stay plain, because a dictionary trained on other record forms saves less than the 113 bytes that name it. ## 6. After the run Two readings were taken after the rule's result was known. The rule does not read them, and they change no verdict. **Without the pin.** A dictionary fixed by a registration, rather than pinned per object, would not pay the pin. With the pin's bytes taken out, the largest nets are 11.1% (`code`, level 19, 16 KiB), 9.9% (`notes`, level 19, 4 KiB), 7.2% (`records`, level 9, 4 KiB) and 2.9% (`spec`, level 19, 16 KiB). `code` alone reaches 10 %, at levels 9 and 19 and not at level 3, so no family would be GO. **`records`, split by object.** The rule's split keeps a record's near-duplicates out of training. Split by object instead, a held-out record's form is usually one the dictionary has seen (735 objects train, 275 held out): | family | level | plain bytes | dictionary | total with the dictionary | net | median saving | break-even, objects | one plain payload | |---|---|---|---|---|---|---|---|---| | records, by object | 3 | 191,574 | 16 KiB | 139,707 | +27.1% | 185 | 66 | 58,432 | | records, by object | 9 | 186,990 | 16 KiB | 135,443 | +27.6% | 188 | 66 | 53,678 | | records, by object | 19 | 183,783 | 16 KiB | 131,681 | +28.3% | 188 | 66 | 50,846 | Within record forms it has seen, then, a dictionary does save: a net of 28.3%, with the pin charged, and it would pass the rule's bar at every level. The rule was written to measure a dictionary against forms it has not met, and it stays as written. Two limits go with this reading: the dictionary has been trained on the forms it is scored on, and objects a sink fetches together can be coded as one plain payload, which here takes 50,846 bytes at level 19, less than the dictionary's total. ## 7. What this does not show - **Traffic.** The workload is one sink fetching a family's held-out objects once, and no sink's fetches were observed. The workload's size decides the verdict in both directions. A sink that fetches fewer objects than the break-even loses bytes. One that fetches more nets more, because the dictionary is charged once: with objects of the held-out mean, the net tends to the mean saving. `code` at level 19 and 16 KiB reaches 10 % at 181 objects, against the 101 held out, and at level 9 at 279. At level 3 it takes the largest dictionary and 1,155 objects, where the net tends to 12.5%, and `notes` reaches every level at 11,162 objects, against 77 held out. `spec` and `records` do not reach 10 % at level 3 at any count: their mean saving there is at most 6.5% and 4.4% of plain. So the verdict is this workload's; a sink that fetches objects of one family by the thousand is a workload the rule does not ask about. - **Text other than this repository's.** The families are English technical prose, Rust source and test records. A corpus of another kind can give another result. - **A stream of one record form.** The corpus holds many small record forms. A long stream of one form was not measured; §6's by-object reading is the nearest thing to it here. - **Time and memory.** Neither was measured. - **Other coders.** Only Zstandard was run, at three levels. ## 8. Verify ```sh python3 -I research/text-dictionary/measure.py check research/text-dictionary/result.json research/text-dictionary.md python3 -I research/text-dictionary/measure.py selftest research/text-dictionary/result.json python3 -I research/text-dictionary/measure.py run --out /tmp/again.json # Python 3.14 or later ``` `check` holds the result to what it cannot state for itself: `rule.json`, the archive and split of `corpus.json`, and each dictionary's length and digest in [`text-dictionary/dictionaries.tar.xz`](text-dictionary/dictionaries.tar.xz), which holds every dictionary the result names. It then recomputes every total from the per-object lengths, each by a formula that shares nothing with the one `run` wrote it with, applies the rule, and reads every marker on this page. The per-object lengths are `run`'s own: `check` has no Zstandard, so only `run` codes an object. `selftest` changes copies of the result, a total with the dictionary's bytes left out, dictionaries recorded as one byte with every total recounted, a lowered rule and another split among them, and requires `check` to refuse each. `run` trains and codes everything again, checks the result before it writes anything, and on a finding exits 1 with the files as they were, which `selftest` also requires. Two runs on one library give one file. The wai-conformance step `text-dictionary` runs the first two.