open-standards/wai/research/text-dictionary.md

A text dictionary, measured against plain Zstandard

Status. A measured record and its procedure, informative. Every figure below is read from text-dictionary/result.json, and research/text-dictionary/measure.py check, which needs no Zstandard, holds the result to the committed rule, split and dictionaries, recomputes every total from the per-object lengths and checks every figure this page quotes (each is followed by a txd marker naming its place in the result). Measured on 6 November 2026. The record decides whether a dictionary-coded text capability is worth registering, and §5 says it is not: no capability, vector or second implementation follows from it.

1. The question

wai.text.zstd codes each object alone, with no dictionary form (SPEC §5). A Zstandard dictionary (RFC 8878 §5) lets a small object refer to text a sink already holds. That helps only if the object saves more than the dictionary costs, so the question is the one prior-carriage §17.2 asks of any set: do the bytes it saves a sink exceed what holding it costs that sink? A dictionary's bytes are counted in full for every sink that fetches it, never divided among the sinks a publisher serves.

2. Corpus

The corpus is this repository's own text, under wai/LICENSE, copied into text-dictionary/corpus.tar.xz with make_corpus.py; corpus.json lists each object's source, its SHA-256 and its side of the split. Four families, each object one payload a sink could fetch:

• notes: the changelog's entries, one object each.
• spec: the documents of wai/extensions/.
• code: the source files of wai/wai-rs/src/ of at most 32 KiB.
• records: the JSON files of at most 8 KiB in seven conformance corpora.

An object whose bytes repeat an earlier object of its family is dropped. No held-out object trains a dictionary. notes is split by time, the newest quarter held out, as a dictionary meets what is written after it. spec and code are split by the digest of the path. records is split by the digest of the directory, so that a held-out record's near-duplicates are held out with it and the dictionary meets record forms it has not seen.

family	objects, train	bytes, train	objects, held out	bytes, held out
notes	234	868,007	77	231,828
spec	78	3,145,397	27	1,571,209
code	306	3,883,202	101	1,184,061
records	636	916,866	374	877,014

3. The rule, and when it was fixed

text-dictionary/rule.json, the corpus, its split and measure.py were committed together before the first measurement, and the commit that holds the result follows them. The rule:

• Levels and sizes. Levels 3, 9 and 19, and dictionaries of 4, 16, 64 and 112 KiB. Plain and dictionary forms of an object are coded at the same level, with no checksum and no dictionary id in the frame.
• Training. One training a family and size, on the train split only, finalised per level.
• Charges. The dictionary's whole length, once for the sink. Each object coded with it also pays 113 bytes, a minimal SPEC §3.1 prior in its manifest, which a plain object does not carry. An object whose dictionary form, with its pin, is not smaller than its plain form is sent plain and saves nothing (prior-carriage §17.2 step 1).
• Workload. One sink fetches every held-out object of the family once.
• Passing. A family, a level and a size pass when the net saving is at least 10 % of the family's plain bytes and the median object's saving is above 0. A family is GO when at every level some size passes. The families that are GO justify a capability, and when none is GO the question is closed.

10 % is the same bar as a catalogue prior's (catalogue-prior §2): a capability costs a registration, a decoder in every sink and vectors, and a saving below that does not pay for them.

4. Results

The run read Python 3.15.2's compression.zstd over libzstd 1.5.7, on arm64 macOS, and a run of Python 3.15.0 over the same library on x86_64 Linux gave every dictionary and every length the same. Every dictionary form was decoded back to its object. Each row gives the dictionary size with the largest net at that level. Total is what the sink moves: each object in its smaller form, then the dictionary. Net is plain less total, as a share of plain. Break-even is the dictionary's bytes over the mean saving per object. One plain payload is the same objects coded together as one plain object, for scale; the rule does not read it.

family	level	plain bytes	dictionary	total with the dictionary	net	median saving	break-even, objects	one plain payload
notes	3	103,158	4 KiB	103,796	-0.6%	28	91	80,368
notes	9	100,257	4 KiB	98,734	+1.5%	63	56	73,645
notes	19	99,138	4 KiB	97,460	+1.7%	64	55	69,419
spec	3	554,829	16 KiB	549,190	+1.0%	960	20	510,830
spec	9	515,680	16 KiB	505,710	+1.9%	1130	17	460,909
spec	19	492,599	16 KiB	481,372	+2.3%	1192	16	418,576
code	3	402,728	16 KiB	388,049	+3.6%	303	53	328,271
code	9	379,006	16 KiB	351,549	+7.2%	420	38	292,188
code	19	369,072	16 KiB	339,365	+8.0%	450	36	266,732
records	3	315,138	4 KiB	314,108	+0.3%	0	299	57,762
records	9	306,960	4 KiB	305,481	+0.5%	0	275	52,278
records	19	299,228	4 KiB	297,638	+0.5%	0	269	49,968

5. Verdict against the rule

No family is GO, at any level. The largest net is 8.0% (code, level 19, 16 KiB), then 2.3% (spec), 1.7% (notes) and 0.5% (records). So a dictionary-coded text capability is not registered, and this record closes the question on this corpus and this workload (§7).

What the figures show:

1. The dictionary's own bytes are the cost. Each family's best size is its smallest or next smallest: at 64 and 112 KiB the median object still saves in every family but records, and the net is below zero in every family at level 3.
2. Large objects have little to gain. A spec document already holds its own repetition: its level-19 median saving at 16 KiB is 1192 bytes on a plain form of tens of kilobytes.
3. Small objects meet the pin. The records median object saves nothing: most held-out records stay plain, because a dictionary trained on other record forms saves less than the 113 bytes that name it.

6. After the run

Two readings were taken after the rule's result was known. The rule does not read them, and they change no verdict.

Without the pin. A dictionary fixed by a registration, rather than pinned per object, would not pay the pin. With the pin's bytes taken out, the largest nets are 11.1% (code, level 19, 16 KiB), 9.9% (notes, level 19, 4 KiB), 7.2% (records, level 9, 4 KiB) and 2.9% (spec, level 19, 16 KiB). code alone reaches 10 %, at levels 9 and 19 and not at level 3, so no family would be GO.

records, split by object. The rule's split keeps a record's near-duplicates out of training. Split by object instead, a held-out record's form is usually one the dictionary has seen (735 objects train, 275 held out):

family	level	plain bytes	dictionary	total with the dictionary	net	median saving	break-even, objects	one plain payload
records, by object	3	191,574	16 KiB	139,707	+27.1%	185	66	58,432
records, by object	9	186,990	16 KiB	135,443	+27.6%	188	66	53,678
records, by object	19	183,783	16 KiB	131,681	+28.3%	188	66	50,846

Within record forms it has seen, then, a dictionary does save: a net of 28.3%, with the pin charged, and it would pass the rule's bar at every level. The rule was written to measure a dictionary against forms it has not met, and it stays as written. Two limits go with this reading: the dictionary has been trained on the forms it is scored on, and objects a sink fetches together can be coded as one plain payload, which here takes 50,846 bytes at level 19, less than the dictionary's total.

7. What this does not show

• Traffic. The workload is one sink fetching a family's held-out objects once, and no sink's fetches were observed. The workload's size decides the verdict in both directions. A sink that fetches fewer objects than the break-even loses bytes. One that fetches more nets more, because the dictionary is charged once: with objects of the held-out mean, the net tends to the mean saving. code at level 19 and 16 KiB reaches 10 % at 181 objects, against the 101 held out, and at level 9 at 279. At level 3 it takes the largest dictionary and 1,155 objects, where the net tends to 12.5%, and notes reaches every level at 11,162 objects, against 77 held out. spec and records do not reach 10 % at level 3 at any count: their mean saving there is at most 6.5% and 4.4% of plain. So the verdict is this workload's; a sink that fetches objects of one family by the thousand is a workload the rule does not ask about.
• Text other than this repository's. The families are English technical prose, Rust source and test records. A corpus of another kind can give another result.
• A stream of one record form. The corpus holds many small record forms. A long stream of one form was not measured; §6's by-object reading is the nearest thing to it here.
• Time and memory. Neither was measured.
• Other coders. Only Zstandard was run, at three levels.

8. Verify

python3 -I research/text-dictionary/measure.py check research/text-dictionary/result.json research/text-dictionary.md
python3 -I research/text-dictionary/measure.py selftest research/text-dictionary/result.json
python3 -I research/text-dictionary/measure.py run --out /tmp/again.json   # Python 3.14 or later

check holds the result to what it cannot state for itself: rule.json, the archive and split of corpus.json, and each dictionary's length and digest in text-dictionary/dictionaries.tar.xz, which holds every dictionary the result names. It then recomputes every total from the per-object lengths, each by a formula that shares nothing with the one run wrote it with, applies the rule, and reads every marker on this page. The per-object lengths are run's own: check has no Zstandard, so only run codes an object.

selftest changes copies of the result, a total with the dictionary's bytes left out, dictionaries recorded as one byte with every total recounted, a lowered rule and another split among them, and requires check to refuse each. run trains and codes everything again, checks the result before it writes anything, and on a finding exits 1 with the files as they were, which selftest also requires. Two runs on one library give one file. The wai-conformance step text-dictionary runs the first two.