open-standards/wai/changes/2026-11-10-every-test-program-written-apart.md
Transaction Science 1d92a45337
All checks were successful
wai-app and Create / wai-app · does this change touch it (push) Successful in 15s
WAI breaking changes and the conformance runner / WAI breaking · does this change touch WAI (push) Successful in 11s
wai key · the real keychain / does this change touch it (push) Successful in 16s
joule-moq-remote host agent / joule-moq-remote · does this change touch it (push) Successful in 20s
joule-moq-room room service / joule-moq-room · does this change touch it (push) Successful in 22s
wai-audit collector and player integration / does this change touch it (push) Successful in 19s
wai-sink-swift native sink package / does this change touch it (push) Successful in 25s
wai-quantum studio and desktop / does this change touch it (push) Successful in 32s
WAI Studio / does this change touch it (push) Successful in 21s
wai-sink-android native sink library / does this change touch it (push) Successful in 59s
wai-quantum for JavaScript / does this change touch it (push) Successful in 1m0s
wai-fleet service / does this change touch it (push) Successful in 1m5s
joule-moq-web browser client / joule-moq-web · does this change touch it (push) Successful in 1m10s
wai-ledger service / does this change touch it (push) Successful in 1m1s
wai key · Windows / does this change touch it (push) Successful in 1m2s
WAI operation catalogue / does this change touch it (push) Successful in 1m7s
wai-app and Create / wai-app · tests, clippy, the line language's verifier (linux) (push) Successful in 55s
WAI breaking changes and the conformance runner / WAI breaking · wire, record and API against main · linux (push) Successful in 1m44s
WAI Studio / the catalogue's actions, the unit tests and the core · linux (push) Successful in 1m20s
wai-audit collector and player integration / collector · linux (push) Successful in 2m6s
wai key · the real keychain / the real keychain · macos (push) Successful in 48s
joule-moq-remote host agent / joule-moq-remote, every target (linux-arm64) (push) Successful in 3m5s
wai-ledger service / service · linux (push) Successful in 2m2s
joule-moq-remote host agent / joule-moq-remote, every target (macos) (push) Successful in 1m4s
wai-audit collector and player integration / collector, player integration and the browser end to end · macos (push) Successful in 1m23s
repository checks / lockfiles, advisories, tests in CI, workflow hygiene, spec and section references, public text (push) Successful in 5m42s
wai-sink-android native sink library / the JNI layer and the platform-free half on the host JVM · linux-arm64 (push) Successful in 4m30s
wai-fleet service / service · linux (push) Successful in 3m14s
joule-moq-room room service / joule-moq-room, every target (push) Successful in 6m14s
wai-quantum studio and desktop / studio · browser (push) Successful in 6m12s
WAI breaking changes and the conformance runner / WAI breaking and conform · their tests, and the independent verifiers through wai conform · linux (push) Successful in 6m48s
wai-quantum for JavaScript / javascript · linux (push) Successful in 5m25s
wai-quantum studio and desktop / desktop app · linux (push) Successful in 9m2s
wai-quantum studio and desktop / desktop app · macos (push) Successful in 4m49s
WAI operation catalogue / the catalogue and every door · linux (push) Successful in 7m52s
wai-app and Create / wai-app · Create, browser end-to-end (macos) (push) Successful in 10m44s
wai-ledger service / service · macos (push) Successful in 56s
wai-fleet service / service · macos (push) Successful in 1m29s
WAI Studio / Studio in Chromium and Firefox · macos (push) Successful in 8m18s
WAI Studio / Studio in WebKit · linux (push) Successful in 12m35s
wai-sink-android native sink library / every ABI, the AAR, and the instrumented tests in a headless emulator · macos (push) Successful in 7m12s
wai-sink-swift native sink package / libwai slices, desktop tests, simulator tests · macos (push) Successful in 16m13s
joule-moq-web browser client / joule-moq-web · browser end-to-end (macos) (push) Successful in 16m3s
wai-quantum studio and desktop / desktop app · windows (push) Successful in 10m17s
wai key · Windows / wai key and the toolchain's tests · windows (push) Successful in 2m32s
wai-rs tests: every program a test runs is written by a process of its own
dx::target's a_kept_key_goes_to_the_keychain_on_standard_input_alone failed on main (CI run
27147, e7c7b209b): "the keychain tool did not start: .../security: Text file busy (os error
26)". It wrote its stand-in `security` from the test process and the keychain ran it at once. A
fork copies the forking process's descriptors into its child until that child's exec, so a spawn
another test's thread made while the file was open for writing still held it open when the
script ran, and Linux refused to run it. Renaming the file into place would not help: the child
holds the same file.

tests/support/program.rs is now the one way a test makes a program it runs: write_program (`sh`
writes the text and sets mode 755) and copy_program (`cp`). That process has exited before either
returns, so the test process never holds a descriptor for the file. The binary's unit tests
include it (src/bin/wai.rs, cfg(test)) and so do the integration tests. Every test that wrote or
copied a program in its own process and then ran it now uses it:
- unit: dx::target's keychain test (the failure), dx::dev's stale-stack programs, dx::doctor's
  automation-package browser; key.rs's and doctor.rs's own sh writers, which already wrote
  this way, now share it;
- integration: cli_project (the refusing packer, and the copy of wai run alone), cli_doctor
  (program(), the automation browser, the copy of wai when a hard link fails), cli_ops (the
  stack programs, the corpus generator), cli_conform_relay (the interop client, the corpus
  tools), cli_target (the old room service), cli_studio (the copy of fake-stack.py), and
  cli_conform's fake_python, which already wrote this way.
The scripts' bytes and what each test asserts are unchanged.

No retry is added. doctor::spawn already retries a spawn that fails with ETXTBSY (20 more tries,
10 ms apart) for the programs wai doctor and wai dev run, which a person can replace on disk
while they start; a_program_still_open_for_writing_is_tried_again holds it. The keychain's only
tool outside tests is /usr/bin/security on macOS, which nothing writes, so a retry there would be
code only a test reaches, and the tests no longer cause the error.

Measured on the Framework (Linux x86_64, 32 cores, shared), the release unit-test binary of wai
(--no-default-features --features quickstart, 172 tests), 8 copies at a time with --test-threads
32, each in its own TMPDIR, both trees at the same time:
  origin/main c4aa4e5d0: 8 of 3000 runs failed, each this test with ETXTBSY (5 of 1000 earlier)
  this change:           0 of 3000 (0 of 1000 earlier)
The binary's 172 tests and cli_doctor, cli_key, cli_ops, cli_project, cli_studio, cli_target,
cli_toolchain, cli_conform and cli_conform_relay pass on Linux and on macOS, the real keychain
test among them on macOS. clippy --all-targets (default features on macOS; full,key_discovery on
both) finds nothing new: the same findings before and after, none in a line changed here.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-09 22:50:40 -05:00

882 B

Every program a test runs is written by a process of its own

  • One helper, for every test that makes a program and runs it. wai's tests write stand-ins for the tools it runs (the keychain's security, a packer, a room service, a browser, a corpus generator) and copy wai itself under other names. Most were written by the test process. A program another test's thread started in that moment kept the file open for writing until it ran its own program, so the stand-in could not start ("Text file busy"): a test of wai auth's keychain failed so in CI. Every one is now written by sh, or copied by cp (tests/support/program.rs, shared by the binary's own tests and the integration tests), and that process has exited before the test goes on, so nothing holds the file when it runs. What each test asserts is unchanged, and only the tests change.