wai: reader limits that bound memory (manifest values, text decode), and the codec wrappers in CI #231
Loading…
Reference in a new issue
No description provided.
Delete branch "wai/security-gaps"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes the two memory gaps
wai/SECURITY.mdlisted, and puts the tests that prove it, and the codec wrappers' fuzz smoke, into CI. Also adds the changelog entry the integer-operator pull requests left out.1. A manifest parse is bounded by its values, not only its length
A manifest at the 2²⁷-byte length limit made of small values took several gigabytes to parse: a parsed value takes more memory than its text.
manifest_too_many_values).check_manifest_text), refuse past the limit, then build each array and object at the size counted and move the members out of that one tree. On a 64-bit target a read allocates at most 128 bytes a value and one a byte: at most 640 MiB for aWAI1manifest at both limits.wai.hgainsMANIFEST_MAX_VALUES; the four stdlib Python verifiers refuse the same manifests; the HTTP corpus gains a case at the limit and one past it for two objects, with a falsification patch and a mutation that must fail them.tests/manifest_parse_memory.rsmeasures each case with a counting allocator;tests/manifest_reader_equivalence.rsholds the new readers to the derived reader's result for every committed envelope and for each with every member removed or retyped. Neither file was named by any workflow step, so no job ran them: both now run in the release step of the cargo-class job.2. zstd and xz decode within a bound
wai.text.zstdandwai.text.xzdecoded the whole stream into memory, so a few kilobytes could ask for any amount.text_decoded_too_long,text_window_too_large.tests/text_decode_bounds.rschecks each limit at its edge, on streams made in the test that sit at the limit and one step past it;tests/text_decode_memory.rsmeasures that refusing a payload that would expand to four times a limit takes no more output buffer than the limit.Planned break for writers. A conforming encoder writes no manifest past the value limit and no text payload past either text limit. No committed envelope or vector is past any of them, and no digest changes.
3. The codec wrappers run in CI
codecsbinds C and C++ libraries no container image of the fleet carries, so no job built it, andtests/fuzz_smoke.rsran only by hand.wai-conformance.ymlgainscodec-wrapperson themacosrunner: fuzz smoke, the text decode limits and the C entry points. Each of those test files is gated on the feature, and a build without it reports "0 passed" and exits 0, so the step fails unless every one of its four binaries ran at least one test.4. One existing test changed
tests/staged_resources.rsbounded a refused layer's peak allocation by twice the envelope's length. Reading a manifest within the new bound takes more than that on a small manifest, so the check failed; and a peak can no longer show a unit decoded after the read, because the read's own peak is higher. The check now measures reading the same envelope and bounds what the refusal allocates beyond it. It was falsified again: allocating one unit's output before the refusal fails it.5. Changelog
Two entries: this work, and the integer-operator crate, bounds pass and certificates, which their own pull requests did not record.
Evidence
Run on the build host; the text checks and the Python verifier at the pushed head:
codec-wrappersstep, exactly as writtenwai_http_vectors verify;http_verify.py; the new patch throughfalsify/run.sh; the new mutation;--check-shards🤖 Generated with Claude Code