ci: every wai-rs test is run by a job or listed with its reason, none can run empty, and wai_prov runs #240

Merged

dcharlot merged 3 commits from ci/tests-are-run into main 2026-10-13 22:27:19 -04:00

Conversation 0 Commits 3 Files changed 7 +611 -10

dcharlot commented 2026-10-13 06:41:18 -04:00

Owner

Copy link

Every integration test of wai-rs is run by a CI job or listed with the reason none runs it, no feature-gated test can run empty, and the provenance tool is built and run.

Why

The conformance workflows name the wai-rs tests they run, one --test NAME at a time. Two failure modes followed, both silent:

• A test file no step names never runs. Of the crate's integration test files, 38 were named by no workflow, among them the integer video and world tests, world_model, world_replay and semantic_path.
• A feature-gated file can run empty. A file that starts #![cfg(feature = "x")], named in a step without x, compiles to a binary with no tests that reports ok. 49 such files had no required-features, so nothing stopped that.

What changes

• scripts/check-tests-in-ci.py (standard library only), with a self-test, in a new workflow tests-in-ci.yml that has no paths: filter and takes seconds. It requires, for wai/wai-rs:
  1. every tests/*.rs is named by a workflow step, or listed with its reason in tests/NOT-IN-CI.txt;
  2. every feature-gated test file has a [[test]] entry whose required-features are exactly its gate, so cargo refuses to build it without them ("target requires the features");
  3. the list names only test files that exist and that no step names, each with a reason;
  4. every [[test]] entry names a file that exists.
• Cargo.toml declares required-features for every feature-gated test file.
• Ten tests join CI:
  • the integer video tests (int_video, int_video_halfpel, int_video_keyframe), the integer world tests (int_world, int_world2) and the JPEG AI bitstream reader (jpegai_bitstream_calibration), in both fast-kernels steps;
  • world_replay, world_model and int_world_shared, in a new step (no other step enables world_model);
  • semantic_path, in the codec-wrappers job, the one place codecs builds.
• Twenty-eight are listed, not run. Each compares the JPEG AI decoder, stage by stage, with values taken from the reference decoder that are not in the tree. On a runner each returns early without checking anything and reports ok, which is worse than not running.
• wai_prov in CI. It needs codecs, so no workflow built it, and its --carry path, which wrote the committed provenance vectors, ran only by hand. codec-wrappers now builds it, carries the committed receipt onto itself, and requires the asset and receipt written again byte for byte. A receipt changed by one byte must be refused. wai-energy-receipts.yml's note that no workflow builds it is updated.

Evidence

Run on the build host:

check	result
all 38 unnamed tests, each with its gate's features	all exit 0; 28 of them only by returning early (their fixtures are absent), hence the list
the fast-kernels step (debug) as written, with the six added tests	exit 0; the added tests also pass in release
the new world step; the codec step; the semantic step	exit 0 each, every binary ran tests
a gated test named without its feature (fuzz_smoke with no codecs)	cargo stops: "target fuzz_smoke in package wai requires the features: codecs"
the provenance step as written	exit 0; with the committed asset changed, it fails
the self-test (nine cases: a covered tree passes; each of eight uncovered trees fails, saying why)	pass
a new unnamed test file added to the real tree	the check names it and fails; removed, it passes
statically, every step that names a test against the features that test now requires (72 named runs)	none lacks a feature
public names (private list), comparative prose, vanity metrics, lockfiles, workflow concurrency	all pass

🤖 Generated with Claude Code

Every integration test of `wai-rs` is run by a CI job or listed with the reason none runs it, no feature-gated test can run empty, and the provenance tool is built and run. ## Why The conformance workflows name the `wai-rs` tests they run, one `--test NAME` at a time. Two failure modes followed, both silent: - **A test file no step names never runs.** Of the crate's integration test files, 38 were named by no workflow, among them the integer video and world tests, `world_model`, `world_replay` and `semantic_path`. - **A feature-gated file can run empty.** A file that starts `#![cfg(feature = "x")]`, named in a step without `x`, compiles to a binary with no tests that reports `ok`. 49 such files had no `required-features`, so nothing stopped that. ## What changes - **`scripts/check-tests-in-ci.py`** (standard library only), with a self-test, in a new workflow `tests-in-ci.yml` that has no `paths:` filter and takes seconds. It requires, for `wai/wai-rs`: 1. every `tests/*.rs` is named by a workflow step, or listed with its reason in `tests/NOT-IN-CI.txt`; 2. every feature-gated test file has a `[[test]]` entry whose `required-features` are exactly its gate, so cargo refuses to build it without them ("target requires the features"); 3. the list names only test files that exist and that no step names, each with a reason; 4. every `[[test]]` entry names a file that exists. - **`Cargo.toml`** declares `required-features` for every feature-gated test file. - **Ten tests join CI:** - the integer video tests (`int_video`, `int_video_halfpel`, `int_video_keyframe`), the integer world tests (`int_world`, `int_world2`) and the JPEG AI bitstream reader (`jpegai_bitstream_calibration`), in both fast-kernels steps; - `world_replay`, `world_model` and `int_world_shared`, in a new step (no other step enables `world_model`); - `semantic_path`, in the `codec-wrappers` job, the one place `codecs` builds. - **Twenty-eight are listed, not run.** Each compares the JPEG AI decoder, stage by stage, with values taken from the reference decoder that are not in the tree. On a runner each returns early without checking anything and reports `ok`, which is worse than not running. - **`wai_prov` in CI.** It needs `codecs`, so no workflow built it, and its `--carry` path, which wrote the committed provenance vectors, ran only by hand. `codec-wrappers` now builds it, carries the committed receipt onto itself, and requires the asset and receipt written again byte for byte. A receipt changed by one byte must be refused. `wai-energy-receipts.yml`'s note that no workflow builds it is updated. ## Evidence Run on the build host: | check | result | |---|---| | all 38 unnamed tests, each with its gate's features | all exit 0; 28 of them only by returning early (their fixtures are absent), hence the list | | the fast-kernels step (debug) as written, with the six added tests | exit 0; the added tests also pass in release | | the new world step; the codec step; the semantic step | exit 0 each, every binary ran tests | | a gated test named without its feature (`fuzz_smoke` with no `codecs`) | cargo stops: "target `fuzz_smoke` in package `wai` requires the features: `codecs`" | | the provenance step as written | exit 0; with the committed asset changed, it fails | | the self-test (nine cases: a covered tree passes; each of eight uncovered trees fails, saying why) | pass | | a new unnamed test file added to the real tree | the check names it and fails; removed, it passes | | statically, every step that names a test against the features that test now requires (72 named runs) | none lacks a feature | | public names (private list), comparative prose, vanity metrics, lockfiles, workflow concurrency | all pass | 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 3 commits 2026-10-13 06:41:19 -04:00

ci: every wai-rs test file is run by a job or listed, and none can run empty ... cdbe206cc7

The conformance workflows name the tests they run. Thirty-eight test files of
wai-rs were named by no step, so no job ran them, and forty-nine feature-gated
ones had no required-features, so a step without the feature would have run
them empty and reported ok.

- scripts/check-tests-in-ci.py, with a self-test, in a new unfiltered workflow:
  every tests/*.rs is named by a step or listed with its reason in
  tests/NOT-IN-CI.txt, and every feature-gated one declares its features.
- Ten real tests join steps: the integer video and world tests and the JPEG AI
  bitstream reader (fast kernels, both builds), world replay, the world model
  and the shared integer world (a new step), the semantic path (codec-wrappers).
- Twenty-eight are listed: each compares the JPEG AI decoder with reference
  values that are not in the tree, and returns without checking when they are
  absent.
- Cargo.toml declares required-features for every feature-gated test.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

ci: build wai_prov and run its carry path against the committed receipt ... b51a356216

wai_prov needs the codecs feature, so no workflow built it and its --carry
path, which wrote the committed provenance vectors, ran only by hand. The
codec-wrappers job builds it on the macOS runner, carries the committed
receipt onto itself and requires the asset and receipt written again byte for
byte, and requires a receipt changed by one byte to be refused.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge main into ci/tests-are-run ... 9e71d1f4fa

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot merged commit 1ec8f44741 into main 2026-10-13 22:27:19 -04:00

dcharlot referenced this pull request from a commit 2026-10-13 22:27:20 -04:00

Merge pull request 'ci: every wai-rs test is run by a job or listed with its reason, none can run empty, and wai_prov runs' (#240) from ci/tests-are-run into main

dcharlot referenced this pull request from a commit 2026-10-13 22:40:05 -04:00

Merge origin/main (at 1ec8f447, with #233, #234, #236, #238, #239, #240) into wai/d2c-decoders-on-ops

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!240

No description provided.