open-standards/wai/cache-conformance

cache-conformance — HTTP caches under http-delivery §9

Checks the cache rules of http-delivery §9 against an HTTP cache that fronts an origin serving the objects of http-conformance/, and reports, per cache label and version, whether the configuration meets each rule.

Provenance. Written here: the tests, the driver and the controls are the code in this directory, and the objects a run fetches are http-conformance's (see its provenance). No media of its own.

file	what
tests.json	the tests: id, rule, level (MUST or informative) and what each checks
driver.py	stages the origin's files and runs the tests against a cache; standard library only, with ../claim-conformance/_blake3.py to check each object against its content hash
mutators.py	seven proxies that each break §9 in one way, the negative controls, and a relay that changes nothing (--mode relay), the positive control through a proxy
run.sh	stages, starts wai http origin, and runs the driver against the origin, each control and each labelled cache

What runs

1. The origin, directly (origin-direct): the reference origin (wai http origin) must pass every MUST test. It answers byte ranges and If-Range, serves a live-edge alias with a shorter max-age, sends §8.4's Cache-Control for the key log, and counts the requests it receives (--count-hits) so the informative tests can tell whether a cache answered from its store.
2. The negative controls (control-flip, control-gzip, control-truncate, control-vary, control-stored, control-ranged, control-reflip): the driver must fail, and must name the tests each defect reaches — flip C1, C3, C11 and C14; gzip C7; truncate C1, C2 and C14; vary C13; stored, which keeps a first answer whole and changes the headers of every later one, C14; ranged, which changes the headers of every 206, C3, C5 and C8; reflip, which keeps a first answer whole and changes a byte of every later one, C1 and C14. A control the driver passes is a defect in the driver.
3. Labelled caches. An environment that provides them lists their labels in <dir>/labels, where <dir> is /opt/wai-cache, or WAI_CACHE_DIR where that is set; each <dir>/<label>/ holds start.sh <port> <origin URL>, which starts the cache in front of the origin, a version file and the config it runs with. This repository holds no cache configuration: the labels are opaque, and a report names a cache only by its label, its package version and the SHA-256 of its configuration. Where /opt/wai-cache names no label, only 1 and 2 run, and the summary says so. A WAI_CACHE_DIR whose labels file is absent or empty fails the run, and a cache that does not start has a row saying so.

A first GET passes through a cache however it treats its store, so C14 asks again once the cache has had a second to store what it fetched, and holds that answer to the origin's bytes and headers. S3 says how many of those repeats the cache answered itself and S4 how many of the run's requests reached the origin. §9 rule 7 lets a cache relay what it cannot store, so the MUST tests pass for a cache that stores nothing; the summary marks a cache whose store answered no repeat as a relay, since such a run says nothing about a store.

Through wai conform

wai conform cache --suite cache --impl '<command>' runs the same tests and writes a signed record of them (conform-result/1, extensions/conformance-results.md). The command starts the cache in the foreground, listening on {port} of 127.0.0.1 in front of {origin}, as a labelled cache's start.sh <port> <origin URL> does; the run stages the origin's files with driver.py stage, starts the reference origin (wai http origin --count-hits), starts the cache, runs driver.py run against it, and stops both. Each test is a case named <id> <name>; a MUST test is required, and an informative one is marked so in the record, reported and not counted in its verdict (§2.3 there; --level all counts it). The relay:

wai conform cache --suite cache --root wai \
  --impl 'python3 -I wai/cache-conformance/mutators.py --mode relay --listen {port} --upstream {origin}'

wai verify --conformant admits a record against this corpus: its digest covers tests.json, the driver, the BLAKE3 the driver checks with, and every file it stages (each object and head proof of http-conformance, and the parameter set wai-rs/tests/vectors/int_mlicpp/model.wmb).

Reports

run.sh <out-dir> writes, under <out-dir>/reports/, report-<label>.json ({cache_label, package_version, config_sha256, results}), requirements-<label>.md (the same as a table, with whether the configuration meets §9), the driver's log per run, and summary.md, which CI also writes to the job summary. The CI job uploads the reports as an artifact.

Running it

cd wai/wai-rs && cargo build --release --no-default-features --features http --bin wai
cd ../.. && WAI=wai/wai-rs/target/release/wai bash wai/cache-conformance/run.sh /tmp/cache-run

With labelled caches in a directory of your own:

WAI=wai/wai-rs/target/release/wai WAI_CACHE_DIR=/path/to/caches bash wai/cache-conformance/run.sh /tmp/cache-run

Python 3 with the standard library is all the driver and the controls need.