f4bbfb229d
measure.py run now checks a result first and writes nothing on a finding, where it used to overwrite result.json and only then exit 1. check takes the rule from rule.json, the archive and split from corpus.json, and each dictionary's length and digest from dictionaries.tar.xz, which is new, so a rule, split or dictionary length rewritten inside a result is refused. The page says the verdict is this workload's, with the object counts at which each family would reach 10 %, and that the median object saves at 64 and 112 KiB in every family but records. Co-Authored-By: Claude Opus 5.5 <[email protected]>
198 lines
16 KiB
Markdown
198 lines
16 KiB
Markdown
# A text dictionary, measured against plain Zstandard
|
|
|
|
> **Status.** A measured record and its procedure, informative. Every figure below is read
|
|
> from [`text-dictionary/result.json`](text-dictionary/result.json), and
|
|
> `research/text-dictionary/measure.py check`, which needs no Zstandard, holds the result to
|
|
> the committed rule, split and dictionaries, recomputes every total from the per-object
|
|
> lengths and checks every figure this page quotes (each is followed by a `txd` marker naming
|
|
> its place in the result). Measured on 6 November 2026.
|
|
> The record decides whether a dictionary-coded text capability is worth registering, and
|
|
> §5 says it is not: no capability, vector or second implementation follows from it.
|
|
|
|
## 1. The question
|
|
|
|
`wai.text.zstd` codes each object alone, with no dictionary form (SPEC §5). A Zstandard
|
|
dictionary (RFC 8878 §5) lets a small object refer to text a sink already holds. That helps
|
|
only if the object saves more than the dictionary costs, so the question is the one
|
|
[prior-carriage](../extensions/prior-carriage.md) §17.2 asks of any set: do the bytes it saves
|
|
a sink exceed what holding it costs that sink? A dictionary's bytes are counted in full for
|
|
every sink that fetches it, never divided among the sinks a publisher serves.
|
|
|
|
## 2. Corpus
|
|
|
|
The corpus is this repository's own text, under wai/LICENSE, copied into
|
|
[`text-dictionary/corpus.tar.xz`](text-dictionary/corpus.tar.xz) with
|
|
[`make_corpus.py`](text-dictionary/make_corpus.py); `corpus.json` lists each object's source,
|
|
its SHA-256 and its side of the split. Four families, each object one payload a sink could
|
|
fetch:
|
|
|
|
- **`notes`**: the changelog's entries, one object each.
|
|
- **`spec`**: the documents of `wai/extensions/`.
|
|
- **`code`**: the source files of `wai/wai-rs/src/` of at most 32 KiB.
|
|
- **`records`**: the JSON files of at most 8 KiB in seven conformance corpora.
|
|
|
|
An object whose bytes repeat an earlier object of its family is dropped. No held-out object
|
|
trains a dictionary. `notes` is split by time, the newest quarter held out, as a dictionary
|
|
meets what is written after it. `spec` and `code` are split by the digest of the path.
|
|
`records` is split by the digest of the directory, so that a held-out record's near-duplicates
|
|
are held out with it and the dictionary meets record forms it has not seen.
|
|
|
|
| family | objects, train | bytes, train | objects, held out | bytes, held out |
|
|
|---|---|---|---|---|
|
|
| `notes` | 234<!--txd:families.notes.objects_train--> | 868,007<!--txd:families.notes.bytes_train--> | 77<!--txd:families.notes.objects_held_out--> | 231,828<!--txd:families.notes.bytes_held_out--> |
|
|
| `spec` | 78<!--txd:families.spec.objects_train--> | 3,145,397<!--txd:families.spec.bytes_train--> | 27<!--txd:families.spec.objects_held_out--> | 1,571,209<!--txd:families.spec.bytes_held_out--> |
|
|
| `code` | 306<!--txd:families.code.objects_train--> | 3,883,202<!--txd:families.code.bytes_train--> | 101<!--txd:families.code.objects_held_out--> | 1,184,061<!--txd:families.code.bytes_held_out--> |
|
|
| `records` | 636<!--txd:families.records.objects_train--> | 916,866<!--txd:families.records.bytes_train--> | 374<!--txd:families.records.objects_held_out--> | 877,014<!--txd:families.records.bytes_held_out--> |
|
|
|
|
## 3. The rule, and when it was fixed
|
|
|
|
[`text-dictionary/rule.json`](text-dictionary/rule.json), the corpus, its split and
|
|
`measure.py` were committed together before the first measurement, and the commit that holds
|
|
the result follows them. The rule:
|
|
|
|
- **Levels and sizes.** Levels 3, 9 and 19, and dictionaries of 4, 16, 64 and 112 KiB. Plain
|
|
and dictionary forms of an object are coded at the same level, with no checksum and no
|
|
dictionary id in the frame.
|
|
- **Training.** One training a family and size, on the train split only, finalised per level.
|
|
- **Charges.** The dictionary's whole length, once for the sink. Each object coded with it
|
|
also pays 113<!--txd:pin_bytes--> bytes, a minimal SPEC §3.1 `prior` in its
|
|
manifest, which a plain object does not carry. An object whose dictionary form, with its
|
|
pin, is not smaller than its plain form is sent plain and saves nothing (prior-carriage
|
|
§17.2 step 1).
|
|
- **Workload.** One sink fetches every held-out object of the family once.
|
|
- **Passing.** A family, a level and a size pass when the net saving is at least 10 % of the
|
|
family's plain bytes and the median object's saving is above 0. A family is GO when at every
|
|
level some size passes. The families that are GO justify a capability, and when none is GO
|
|
the question is closed.
|
|
|
|
10 % is the same bar as a catalogue prior's ([catalogue-prior](catalogue-prior.md) §2): a
|
|
capability costs a registration, a decoder in every sink and vectors, and a saving below that
|
|
does not pay for them.
|
|
|
|
## 4. Results
|
|
|
|
The run read Python 3.15.2's `compression.zstd` over libzstd 1.5.7, on arm64 macOS, and a run of
|
|
Python 3.15.0 over the same library on x86_64 Linux gave every dictionary and every length the
|
|
same. Every dictionary form was decoded back to its object. Each row gives the dictionary size
|
|
with the largest net at that level. *Total* is what the sink moves: each
|
|
object in its smaller form, then the dictionary. *Net* is plain less total, as a share of plain.
|
|
*Break-even* is the dictionary's bytes over the mean saving per object. *One plain payload* is
|
|
the same objects coded together as one plain object, for scale; the rule does not read it.
|
|
|
|
| family | level | plain bytes | dictionary | total with the dictionary | net | median saving | break-even, objects | one plain payload |
|
|
|---|---|---|---|---|---|---|---|---|
|
|
| notes | 3 | 103,158<!--txd:cell:notes:3:4096.plain_total--> | 4 KiB | 103,796<!--txd:cell:notes:3:4096.total--> | -0.6%<!--txd:cell:notes:3:4096.net_fraction--> | 28<!--txd:cell:notes:3:4096.median_saving--> | 91<!--txd:cell:notes:3:4096.break_even_objects--> | 80,368<!--txd:families.notes.solid.3--> |
|
|
| notes | 9 | 100,257<!--txd:cell:notes:9:4096.plain_total--> | 4 KiB | 98,734<!--txd:cell:notes:9:4096.total--> | +1.5%<!--txd:cell:notes:9:4096.net_fraction--> | 63<!--txd:cell:notes:9:4096.median_saving--> | 56<!--txd:cell:notes:9:4096.break_even_objects--> | 73,645<!--txd:families.notes.solid.9--> |
|
|
| notes | 19 | 99,138<!--txd:cell:notes:19:4096.plain_total--> | 4 KiB | 97,460<!--txd:cell:notes:19:4096.total--> | +1.7%<!--txd:cell:notes:19:4096.net_fraction--> | 64<!--txd:cell:notes:19:4096.median_saving--> | 55<!--txd:cell:notes:19:4096.break_even_objects--> | 69,419<!--txd:families.notes.solid.19--> |
|
|
| spec | 3 | 554,829<!--txd:cell:spec:3:16384.plain_total--> | 16 KiB | 549,190<!--txd:cell:spec:3:16384.total--> | +1.0%<!--txd:cell:spec:3:16384.net_fraction--> | 960<!--txd:cell:spec:3:16384.median_saving--> | 20<!--txd:cell:spec:3:16384.break_even_objects--> | 510,830<!--txd:families.spec.solid.3--> |
|
|
| spec | 9 | 515,680<!--txd:cell:spec:9:16384.plain_total--> | 16 KiB | 505,710<!--txd:cell:spec:9:16384.total--> | +1.9%<!--txd:cell:spec:9:16384.net_fraction--> | 1130<!--txd:cell:spec:9:16384.median_saving--> | 17<!--txd:cell:spec:9:16384.break_even_objects--> | 460,909<!--txd:families.spec.solid.9--> |
|
|
| spec | 19 | 492,599<!--txd:cell:spec:19:16384.plain_total--> | 16 KiB | 481,372<!--txd:cell:spec:19:16384.total--> | +2.3%<!--txd:cell:spec:19:16384.net_fraction--> | 1192<!--txd:cell:spec:19:16384.median_saving--> | 16<!--txd:cell:spec:19:16384.break_even_objects--> | 418,576<!--txd:families.spec.solid.19--> |
|
|
| code | 3 | 402,728<!--txd:cell:code:3:16384.plain_total--> | 16 KiB | 388,049<!--txd:cell:code:3:16384.total--> | +3.6%<!--txd:cell:code:3:16384.net_fraction--> | 303<!--txd:cell:code:3:16384.median_saving--> | 53<!--txd:cell:code:3:16384.break_even_objects--> | 328,271<!--txd:families.code.solid.3--> |
|
|
| code | 9 | 379,006<!--txd:cell:code:9:16384.plain_total--> | 16 KiB | 351,549<!--txd:cell:code:9:16384.total--> | +7.2%<!--txd:cell:code:9:16384.net_fraction--> | 420<!--txd:cell:code:9:16384.median_saving--> | 38<!--txd:cell:code:9:16384.break_even_objects--> | 292,188<!--txd:families.code.solid.9--> |
|
|
| code | 19 | 369,072<!--txd:cell:code:19:16384.plain_total--> | 16 KiB | 339,365<!--txd:cell:code:19:16384.total--> | +8.0%<!--txd:cell:code:19:16384.net_fraction--> | 450<!--txd:cell:code:19:16384.median_saving--> | 36<!--txd:cell:code:19:16384.break_even_objects--> | 266,732<!--txd:families.code.solid.19--> |
|
|
| records | 3 | 315,138<!--txd:cell:records:3:4096.plain_total--> | 4 KiB | 314,108<!--txd:cell:records:3:4096.total--> | +0.3%<!--txd:cell:records:3:4096.net_fraction--> | 0<!--txd:cell:records:3:4096.median_saving--> | 299<!--txd:cell:records:3:4096.break_even_objects--> | 57,762<!--txd:families.records.solid.3--> |
|
|
| records | 9 | 306,960<!--txd:cell:records:9:4096.plain_total--> | 4 KiB | 305,481<!--txd:cell:records:9:4096.total--> | +0.5%<!--txd:cell:records:9:4096.net_fraction--> | 0<!--txd:cell:records:9:4096.median_saving--> | 275<!--txd:cell:records:9:4096.break_even_objects--> | 52,278<!--txd:families.records.solid.9--> |
|
|
| records | 19 | 299,228<!--txd:cell:records:19:4096.plain_total--> | 4 KiB | 297,638<!--txd:cell:records:19:4096.total--> | +0.5%<!--txd:cell:records:19:4096.net_fraction--> | 0<!--txd:cell:records:19:4096.median_saving--> | 269<!--txd:cell:records:19:4096.break_even_objects--> | 49,968<!--txd:families.records.solid.19--> |
|
|
|
|
## 5. Verdict against the rule
|
|
|
|
No family is GO, at any level. The largest net is 8.0%<!--txd:cell:code:19:16384.net_fraction-->
|
|
(`code`, level 19, 16 KiB), then 2.3%<!--txd:cell:spec:19:16384.net_fraction--> (`spec`),
|
|
1.7%<!--txd:cell:notes:19:4096.net_fraction--> (`notes`) and
|
|
0.5%<!--txd:cell:records:19:4096.net_fraction--> (`records`). So a dictionary-coded text
|
|
capability is not registered, and this record closes the question on this corpus and this
|
|
workload (§7).
|
|
|
|
What the figures show:
|
|
|
|
1. **The dictionary's own bytes are the cost.** Each family's best size is its smallest or next
|
|
smallest: at 64 and 112 KiB the median object still saves in every family but `records`, and
|
|
the net is below zero in every family at level 3.
|
|
2. **Large objects have little to gain.** A `spec` document already holds its own repetition:
|
|
its level-19 median saving at 16 KiB is 1192<!--txd:cell:spec:19:16384.median_saving-->
|
|
bytes on a plain form of tens of kilobytes.
|
|
3. **Small objects meet the pin.** The `records` median object saves nothing: most held-out
|
|
records stay plain, because a dictionary trained on other record forms saves less than the
|
|
113<!--txd:pin_bytes--> bytes that name it.
|
|
|
|
## 6. After the run
|
|
|
|
Two readings were taken after the rule's result was known. The rule does not read them, and
|
|
they change no verdict.
|
|
|
|
**Without the pin.** A dictionary fixed by a registration, rather than pinned per object, would
|
|
not pay the pin. With the pin's bytes taken out, the largest nets are
|
|
11.1%<!--txd:cell:code:19:16384.net_fraction_no_pin--> (`code`, level 19, 16 KiB),
|
|
9.9%<!--txd:cell:notes:19:4096.net_fraction_no_pin--> (`notes`, level 19, 4 KiB),
|
|
7.2%<!--txd:cell:records:9:4096.net_fraction_no_pin--> (`records`, level 9, 4 KiB) and
|
|
2.9%<!--txd:cell:spec:19:16384.net_fraction_no_pin--> (`spec`, level 19, 16 KiB).
|
|
`code` alone reaches 10 %, at levels 9 and 19 and not at level 3, so no family would be GO.
|
|
|
|
**`records`, split by object.** The rule's split keeps a record's near-duplicates out of
|
|
training. Split by object instead, a held-out record's form is usually one the dictionary has
|
|
seen (735<!--txd:after_the_run.records_by_object.objects_train--> objects train,
|
|
275<!--txd:after_the_run.records_by_object.objects_held_out--> held out):
|
|
|
|
| family | level | plain bytes | dictionary | total with the dictionary | net | median saving | break-even, objects | one plain payload |
|
|
|---|---|---|---|---|---|---|---|---|
|
|
| records, by object | 3 | 191,574<!--txd:cell:records_by_object:3:16384.plain_total--> | 16 KiB | 139,707<!--txd:cell:records_by_object:3:16384.total--> | +27.1%<!--txd:cell:records_by_object:3:16384.net_fraction--> | 185<!--txd:cell:records_by_object:3:16384.median_saving--> | 66<!--txd:cell:records_by_object:3:16384.break_even_objects--> | 58,432<!--txd:after_the_run.records_by_object.solid.3--> |
|
|
| records, by object | 9 | 186,990<!--txd:cell:records_by_object:9:16384.plain_total--> | 16 KiB | 135,443<!--txd:cell:records_by_object:9:16384.total--> | +27.6%<!--txd:cell:records_by_object:9:16384.net_fraction--> | 188<!--txd:cell:records_by_object:9:16384.median_saving--> | 66<!--txd:cell:records_by_object:9:16384.break_even_objects--> | 53,678<!--txd:after_the_run.records_by_object.solid.9--> |
|
|
| records, by object | 19 | 183,783<!--txd:cell:records_by_object:19:16384.plain_total--> | 16 KiB | 131,681<!--txd:cell:records_by_object:19:16384.total--> | +28.3%<!--txd:cell:records_by_object:19:16384.net_fraction--> | 188<!--txd:cell:records_by_object:19:16384.median_saving--> | 66<!--txd:cell:records_by_object:19:16384.break_even_objects--> | 50,846<!--txd:after_the_run.records_by_object.solid.19--> |
|
|
|
|
Within record forms it has seen, then, a dictionary does save: a net of
|
|
28.3%<!--txd:cell:records_by_object:19:16384.net_fraction-->, with the pin charged, and
|
|
it would pass the rule's bar at every level. The rule was written to measure a dictionary
|
|
against forms it has not met, and it stays as written. Two limits go with this reading: the
|
|
dictionary has been trained on the forms it is scored on, and objects a sink fetches together
|
|
can be coded as one plain payload, which here takes
|
|
50,846<!--txd:after_the_run.records_by_object.solid.19--> bytes at level 19,
|
|
less than the dictionary's total.
|
|
|
|
## 7. What this does not show
|
|
|
|
- **Traffic.** The workload is one sink fetching a family's held-out objects once, and no sink's
|
|
fetches were observed. The workload's size decides the verdict in both directions. A sink that
|
|
fetches fewer objects than the break-even loses bytes. One that fetches more nets more, because
|
|
the dictionary is charged once: with objects of the held-out mean, the net tends to the mean
|
|
saving. `code` at level 19 and 16 KiB reaches 10 % at
|
|
181<!--txd:cell:code:19:16384.objects_to_reach_min--> objects, against the
|
|
101<!--txd:families.code.objects_held_out--> held out, and at level 9 at
|
|
279<!--txd:cell:code:9:16384.objects_to_reach_min-->. At level 3 it takes the largest
|
|
dictionary and 1,155<!--txd:cell:code:3:114688.objects_to_reach_min--> objects, where the net
|
|
tends to 12.5%<!--txd:cell:code:3:114688.saving_fraction-->, and `notes` reaches every level
|
|
at 11,162<!--txd:cell:notes:3:114688.objects_to_reach_min--> objects, against
|
|
77<!--txd:families.notes.objects_held_out--> held out. `spec` and `records` do not reach 10 %
|
|
at level 3 at any count: their mean saving there is at most
|
|
6.5%<!--txd:cell:spec:3:114688.saving_fraction--> and
|
|
4.4%<!--txd:cell:records:3:65536.saving_fraction--> of plain. So the verdict is this
|
|
workload's; a sink that fetches objects of one family by the thousand is a workload the rule
|
|
does not ask about.
|
|
- **Text other than this repository's.** The families are English technical prose, Rust source
|
|
and test records. A corpus of another kind can give another result.
|
|
- **A stream of one record form.** The corpus holds many small record forms. A long stream of
|
|
one form was not measured; §6's by-object reading is the nearest thing to it here.
|
|
- **Time and memory.** Neither was measured.
|
|
- **Other coders.** Only Zstandard was run, at three levels.
|
|
|
|
## 8. Verify
|
|
|
|
```sh
|
|
python3 -I research/text-dictionary/measure.py check research/text-dictionary/result.json research/text-dictionary.md
|
|
python3 -I research/text-dictionary/measure.py selftest research/text-dictionary/result.json
|
|
python3 -I research/text-dictionary/measure.py run --out /tmp/again.json # Python 3.14 or later
|
|
```
|
|
|
|
`check` holds the result to what it cannot state for itself: `rule.json`, the archive and split
|
|
of `corpus.json`, and each dictionary's length and digest in
|
|
[`text-dictionary/dictionaries.tar.xz`](text-dictionary/dictionaries.tar.xz), which holds every
|
|
dictionary the result names. It then recomputes every total from the per-object lengths, each by
|
|
a formula that shares nothing with the one `run` wrote it with, applies the rule, and reads every
|
|
marker on this page. The per-object lengths are `run`'s own: `check` has no Zstandard, so only
|
|
`run` codes an object.
|
|
|
|
`selftest` changes copies of the result, a total with the dictionary's bytes left out, dictionaries
|
|
recorded as one byte with every total recounted, a lowered rule and another split among them, and
|
|
requires `check` to refuse each. `run` trains and codes everything again, checks the result before
|
|
it writes anything, and on a finding exits 1 with the files as they were, which `selftest` also
|
|
requires. Two runs on one library give one file. The wai-conformance step `text-dictionary` runs
|
|
the first two.
|