ci: the wai-quantum legs build the commit they test, and fail when they did not #666

Merged

dcharlot merged 2 commits from ci/quantum-archive-mtime into main 2026-11-06 14:42:47 -05:00

Conversation 0 Commits 2 Files changed 7 +93 -8

dcharlot commented 2026-11-06 03:23:54 -05:00

Owner

Copy link

What was wrong

Seven wai-quantum workflows (processors, dsp, dataflow, sme, tensor-engine, tensor-program, vector) fetch the forge's archive of the commit, extract it to the same path in every job container, and build into a target directory kept in the runner's /ci-cache. tar gave every file its commit's time, and cargo takes a crate as built when no source file is newer than its last build. So a leg whose commit was older than the cache's last build ran the binary another commit had built there, and its result said nothing about its own code.

Measured on 5 Nov (job logs, every emulated leg from 15:00 to 19:00 EST):

• 16 emulated legs went green with no Compiling wai-quantum line, each on a binary from a commit whose wai/wai-quantum differs (1 to 13 files): 6 of #646's 7 finished legs, #645 powerpc64le, #648 i686, #659 aarch64, loongarch64 and s390x, and others.
• #662's three reds (emulated armv7, emulated sparc64, signal processor) were its legs running #660's two frame_reference tests against #662's check for three. #662's code passes on both targets when built fresh in the pinned images.
• Some binaries mixed commits: a leg rebuilt only its test crate (a file listed in the cache's dep-info was missing from its tree) and linked it against another commit's library.

The change

• Fetch the source extracts with tar -xmf: every file gets the time it was extracted.
• That alone does not close it. Each build takes the host-wide flock after its extraction, and legs wait on it for many minutes. Another run can build in the cache meanwhile, and then this run's files are the older again. So every step that builds touches the extracted tree (find ../.. -type f -exec touch {} +) once it holds the lock. That is 10 steps: processors 1, dsp 3 (one per target directory), dataflow 2, sme, tensor-engine, tensor-program's cpu leg and vector's container leg 1 each.
• The guard. The first build in each of those steps must print Compiling wai-quantum v<version>, anchored, so it cannot match wai-quantum-cli, -service or -writer. A build that compiled nothing fails the leg with "the build compiled no wai-quantum: its tests ran a binary already in the cache". Dataflow's encoder build drops -q so its lines are visible.

Cost: each of those steps recompiles wai-quantum (about 35 s to 4 min per leg in the 5 Nov logs), which every leg that tested its own code already paid. Registry dependencies stay cached.

Not changed: the host-runner legs (macOS, linux-hw, linux-amd, linux-arm64). They check out with git into a fresh tree, so their files carry the checkout's time. The hardware legs that share $HOME/hw-target have a narrower form of the same race, through cargo's own lock on the target directory, if two run at once on one host. That is left for a separate change.

No changelog entry: nothing under wai/ changes, and scripts/check-changelog-fragment.py asks for one only then.

An open pull request picks this up only once its branch has it (a pull request runs its head's workflow file), so the open wai-quantum PRs need main merged in after this lands.

The guard's proof

I took wai-quantum-processors.yml's two steps from git, this commit's and main's, unchanged except for the target (native x86_64), the head SHA and the cache path. I ran them on the Linux x86_64 build host against one persistent cache, with no rustup on PATH, as in the job container. X is #660's head and Y is main; both have two frame_reference tests, so a stale binary passes every test-count check and only the guard can tell.

Phase	What ran	Exit	Compiling wai-quantum lines	Result
P1	X, fetched and tested by this commit's steps (the cache's first build)	0	3	fresh: 23/23, 2/2, 167 passed, 1033 filtered
P2	Y fetched with tar -xf and main's step, i.e. main today	0	0	false green: 1033 filtered is X's library, not Y's (P5: 1026)
P3	the same tree, this commit's step without its refresh	1	0	the guard: the build compiled no wai-quantum: its tests ran a binary already in the cache
P4a	Y extracted with -m, then another run builds X before Y holds the lock	0	3	(the other run)
P4b	Y, extracted before that build, this commit's step without its refresh	1	0	the guard again: tar -m alone does not close the race
P5	the same tree, this commit's step as committed	0	3	fresh: 23/23, 2/2, 167 passed, 1026 filtered

Applied to the 5 Nov job logs, the guard's pattern fails #662's three stale jobs (86809, 86815, 86773) and passes the fresh ones (86810, 86631).

Also checked locally: all seven files parse, and every changed step goes lock, refresh, first build, guard, in that order. check-workflow-concurrency.sh, check-wasmtime-pin.py and check-tests-in-ci.py pass. The branch merges cleanly with #662, the one open pull request that edits these files.

🤖 Generated with Claude Code

## What was wrong Seven wai-quantum workflows (processors, dsp, dataflow, sme, tensor-engine, tensor-program, vector) fetch the forge's archive of the commit, extract it to the same path in every job container, and build into a target directory kept in the runner's `/ci-cache`. `tar` gave every file its commit's time, and cargo takes a crate as built when no source file is newer than its last build. So a leg whose commit was older than the cache's last build **ran the binary another commit had built there**, and its result said nothing about its own code. Measured on 5 Nov (job logs, every emulated leg from 15:00 to 19:00 EST): - 16 emulated legs went green with no `Compiling wai-quantum` line, each on a binary from a commit whose `wai/wai-quantum` differs (1 to 13 files): 6 of #646's 7 finished legs, #645 powerpc64le, #648 i686, #659 aarch64, loongarch64 and s390x, and others. - #662's three reds (emulated armv7, emulated sparc64, signal processor) were its legs running #660's two `frame_reference` tests against #662's check for three. #662's code passes on both targets when built fresh in the pinned images. - Some binaries mixed commits: a leg rebuilt only its test crate (a file listed in the cache's dep-info was missing from its tree) and linked it against another commit's library. ## The change - **Fetch the source** extracts with `tar -xmf`: every file gets the time it was extracted. - **That alone does not close it.** Each build takes the host-wide `flock` after its extraction, and legs wait on it for many minutes. Another run can build in the cache meanwhile, and then this run's files are the older again. So every step that builds touches the extracted tree (`find ../.. -type f -exec touch {} +`) once it holds the lock. That is 10 steps: processors 1, dsp 3 (one per target directory), dataflow 2, sme, tensor-engine, tensor-program's cpu leg and vector's container leg 1 each. - **The guard.** The first build in each of those steps must print `Compiling wai-quantum v<version>`, anchored, so it cannot match wai-quantum-cli, -service or -writer. A build that compiled nothing fails the leg with "the build compiled no wai-quantum: its tests ran a binary already in the cache". Dataflow's encoder build drops `-q` so its lines are visible. Cost: each of those steps recompiles wai-quantum (about 35 s to 4 min per leg in the 5 Nov logs), which every leg that tested its own code already paid. Registry dependencies stay cached. Not changed: the host-runner legs (macOS, linux-hw, linux-amd, linux-arm64). They check out with git into a fresh tree, so their files carry the checkout's time. The hardware legs that share `$HOME/hw-target` have a narrower form of the same race, through cargo's own lock on the target directory, if two run at once on one host. That is left for a separate change. No changelog entry: nothing under `wai/` changes, and `scripts/check-changelog-fragment.py` asks for one only then. An open pull request picks this up only once its branch has it (a pull request runs its head's workflow file), so the open wai-quantum PRs need main merged in after this lands. ## The guard's proof I took wai-quantum-processors.yml's two steps from git, this commit's and main's, unchanged except for the target (native x86_64), the head SHA and the cache path. I ran them on the Linux x86_64 build host against one persistent cache, with no rustup on PATH, as in the job container. X is #660's head and Y is main; both have two `frame_reference` tests, so a stale binary passes every test-count check and only the guard can tell. | Phase | What ran | Exit | `Compiling wai-quantum` lines | Result | |---|---|---|---|---| | P1 | X, fetched and tested by this commit's steps (the cache's first build) | 0 | 3 | fresh: 23/23, 2/2, 167 passed, 1033 filtered | | P2 | Y fetched with `tar -xf` and main's step, i.e. main today | **0** | **0** | **false green**: 1033 filtered is X's library, not Y's (P5: 1026) | | P3 | the same tree, this commit's step without its refresh | **1** | 0 | the guard: `the build compiled no wai-quantum: its tests ran a binary already in the cache` | | P4a | Y extracted with `-m`, then another run builds X before Y holds the lock | 0 | 3 | (the other run) | | P4b | Y, extracted before that build, this commit's step without its refresh | **1** | 0 | the guard again: `tar -m` alone does not close the race | | P5 | the same tree, this commit's step as committed | 0 | 3 | fresh: 23/23, 2/2, 167 passed, 1026 filtered | Applied to the 5 Nov job logs, the guard's pattern fails #662's three stale jobs (86809, 86815, 86773) and passes the fresh ones (86810, 86631). Also checked locally: all seven files parse, and every changed step goes lock, refresh, first build, guard, in that order. `check-workflow-concurrency.sh`, `check-wasmtime-pin.py` and `check-tests-in-ci.py` pass. The branch merges cleanly with #662, the one open pull request that edits these files. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 1 commit 2026-11-06 03:23:54 -05:00

ci: the wai-quantum legs build the commit they test, and fail when they did not ... 9c3c42ee28

Seven wai-quantum workflows (processors, dsp, dataflow, sme, tensor-engine,
tensor-program, vector) take the forge's archive of the commit, extract it to
the same path on every run, and build into a target directory kept in the
runner's cache. tar gave every file its commit's time, and cargo takes a crate
as built when no source file is newer than its last build. So a commit older
than the cache's last build ran the binary another commit had built there, and
its tests said nothing about its own code. On 5 Nov, 16 emulated legs went
green on another commit's binary, and #662's three reds were its legs running
#660's two frame_reference tests against #662's check for three.

- Fetch the source extracts with tar -m: every file gets the time it was
  extracted.
- That alone does not close it. Each build waits on the host-wide lock after
  its extraction, and another run can build in the cache meanwhile, after
  which this one's files are again the older. So every step that builds
  touches the extracted tree once it holds the lock.
- The first build in each such step must print "Compiling wai-quantum v...";
  one that compiled nothing fails the leg, saying its tests would run a binary
  already in the cache.

The host-runner legs (macOS, linux-hw, linux-amd, linux-arm64) check out with
git into a fresh tree, and are not changed here.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-11-06 08:45:35 -05:00

ci: the emulated vector and SME legs bring their own emulator #674

dcharlot added 1 commit 2026-11-06 13:54:20 -05:00

Merge main into ci/quantum-archive-mtime: the emulated legs bring their own emulator (#674) ... 628e9923bf

Main now runs the vector and SME emulated legs under the job's pinned QEMU, so
this branch's sve, rvv and SME legs no longer depend on which `linux` runner
claims them. The two changes touch different lines of both workflows.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot merged commit bbc5e531ce into main 2026-11-06 14:42:47 -05:00

dcharlot referenced this pull request from a commit 2026-11-06 14:42:53 -05:00

Merge pull request 'ci: the wai-quantum legs build the commit they test, and fail when they did not' (#666) from ci/quantum-archive-mtime into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!666

No description provided.