From f82d81e5e9b9adcfbb3531f800ad4c3845d97fe6 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Mon, 21 Sep 2026 15:23:39 -0400 Subject: [PATCH 1/3] ci(wai-neural-e2e): a skipped reference comparison is reported as a pass MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every `*_matches_reference` test in `neural_native_decode.rs` opens with if !wai.exists() || !ref.exists() || !model.exists() { eprintln!("skipping ... /tmp artifacts not present"); return; } so a missing fixture makes the test skip — and cargo reports a skip of this shape as a **pass**. Verified by running the job's exact command on a machine with no fixtures: test result: ok. 6 passed; 0 failed Six green, nothing compared. The job is named "synthetic fixtures → Rust decode → reference (±4)" and in that state it verifies none of those things. Today the job is red, so this is latent rather than active: the Python fixture step must be failing outright, which is the one way this surfaces. But the moment fixture generation succeeds while writing nothing — a harness quietly producing 0 of 5, a path change, a partial install — the job goes fully green having checked nothing, and the neural byte-exactness claim loses its only end-to-end guard with no signal at all. The fix asserts on the skip rather than enumerating fixture paths. A path list here would be a second copy of the one in the test file, free to drift from it, which is the same failure one level up. Grepping the run for "skipping" stays correct whatever fixtures the harnesses grow. Falsified: with no fixtures, `cargo test` exits 0 and the guard exits 1, naming all six skipped comparisons. Two things this does NOT do. It does not diagnose the current red — the forge exposes no job logs, so the failing step cannot be identified from here; the most likely cause is the unpinned `pip install torch torchvision compressai onnx onnxruntime`, whose versions have moved since this workflow last ran. And it does not pin those versions, because that would be fixing something I have not diagnosed. Worth noting how rarely this runs: its trigger paths (`wai/wai-rs/src/neural/**`, the test, the validator) were last touched 2026-07-31, and the workflow itself 2026-08-26. A gate that fires a few times a year can rot between firings, and this one was only exercised now because an unrelated stale-branch rebase produced a push delta wide enough to match its paths. Co-Authored-By: Claude Opus 5 --- .github/workflows/wai-neural-e2e.yml | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/.github/workflows/wai-neural-e2e.yml b/.github/workflows/wai-neural-e2e.yml index fe28ce1fe..eec8a48f2 100644 --- a/.github/workflows/wai-neural-e2e.yml +++ b/.github/workflows/wai-neural-e2e.yml @@ -60,4 +60,25 @@ jobs: working-directory: wai/wai-rs # --no-default-features: the neural decoders need only `ort`, NOT the C # codec chain (AV1/JXL/opus), so the job needs no system codec deps. - run: cargo test --no-default-features --features neural --test neural_native_decode -- _matches_reference --nocapture + # + # The grep is the assertion, not decoration. Every `*_matches_reference` + # test opens with `if !wai.exists() || !ref.exists() || !model.exists() { + # eprintln!("skipping ..."); return; }` — so a missing fixture makes the + # test SKIP, and a skip is reported as a PASS. Verified by running this + # exact command on a machine with no /tmp fixtures: it prints + # "6 passed; 0 failed" while comparing nothing. + # + # That means a green `cargo test` here does NOT mean the decoders were + # checked against their references. If the Python fixture step ever + # succeeds while writing nothing, this job goes green having verified + # nothing at all — the failure mode is silent and looks like success. + # Failing on "skipping" is what makes the job's name true. + run: | + set -o pipefail + cargo test --no-default-features --features neural --test neural_native_decode \ + -- _matches_reference --nocapture 2>&1 | tee /tmp/neural_e2e.log + if grep -q "skipping" /tmp/neural_e2e.log; then + echo "::error::a reference comparison was SKIPPED — its fixtures are missing, so this job verified nothing" + grep "skipping" /tmp/neural_e2e.log + exit 1 + fi -- 2.54.0 (Apple Git-157) From 7508665026e2b6c78de87b47b7860fa474606968 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Tue, 22 Sep 2026 01:20:17 -0400 Subject: [PATCH 2/3] wai: the fixture generator reports partial success as success MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The companion to the skip guard, and the reason it is not hypothetical. `wai_validate_synthetic.py` ends with `return 0 if ok else 1`, and its harness loop deliberately catches per-harness exceptions to keep going. So a run where one of five harnesses succeeded exited **0**: the CI step went green, the four decoders whose fixtures were never written then SKIPPED in the Rust suite, and a skip of that shape is reported as a pass. 5/5 -> exit 0 1/5 -> exit 0 0/5 -> exit 1 The job therefore had a path to fully green while having checked one decoder out of five, with nothing in the output saying so. That is the scenario the previous commit's guard was written against; it turns out the generator hands it over rather than it needing an unlikely coincidence. Now `return 0 if ok == len(harnesses) else 1`. Partial generation stays useful interactively — the per-harness ✓/✗ lines print either way and a caller wanting one fixture can ignore the exit code — but CI can no longer read a partial run as a pass. Kept alongside the grep guard rather than replacing it. They fail on different things: this one refuses to claim fixtures that were not written, the guard refuses a comparison that did not happen whatever the cause. On the job's current red, now measured rather than assumed: the failing status reports **1m7s on this branch and 1m5s on #54**, whose tree does not touch this workflow at all. Near-identical, and far too early to have compiled the `neural` feature set and reached the Rust step where the guard lives. Both die at the same earlier Python step, so neither change here causes the red and neither can clear it. Diagnosing it needs the job log, which this forge does not expose; the unpinned `pip install torch torchvision compressai onnx onnxruntime` remains the likely cause and is still not being fixed blind. Co-Authored-By: Claude Opus 5 --- wai/tools/wai_validate_synthetic.py | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/wai/tools/wai_validate_synthetic.py b/wai/tools/wai_validate_synthetic.py index 8f5dc70ab..fbe356212 100644 --- a/wai/tools/wai_validate_synthetic.py +++ b/wai/tools/wai_validate_synthetic.py @@ -397,7 +397,19 @@ def main() -> int: print(f" ✗ {name}: {type(e).__name__}: {e}") print(f"generated {ok}/{len(harnesses)} synthetic neural fixtures into /tmp") print("now: cargo test --features neural --test neural_native_decode -- _matches_reference") - return 0 if ok else 1 + # Every harness must succeed, not merely one of them. + # + # `0 if ok else 1` reported success whenever a single harness worked, and the + # loop above deliberately catches per-harness exceptions to keep going. So a + # 1-of-5 run exited 0, the CI step went green, and the four decoders whose + # fixtures were never written then SKIPPED in the Rust suite — where a skip is + # reported as a pass. The job came out fully green having checked one decoder + # out of five, and nothing in the output said so. + # + # Partial generation is still useful interactively: the per-harness ✓/✗ lines + # are printed either way, and a caller who wants one fixture can ignore the + # exit code. What is not acceptable is CI reading a partial run as a pass. + return 0 if ok == len(harnesses) else 1 if __name__ == "__main__": -- 2.54.0 (Apple Git-157) From 2579c9c42d67582f8670c36c4b6af7d1a1fcf88d Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Tue, 29 Sep 2026 02:20:07 -0400 Subject: [PATCH 3/3] wai-neural-e2e: make the job run, and fix the decoder mismatch it was hiding MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This job has failed at ~1 minute on every run, so its reference comparisons have never executed in CI. Two root causes, both now established from evidence — the first from the job's actual log. 1. The Rust test could not link on the runner. The job reaches its last step and fails there. From the job log (forge 16.0.5 exposes `GET /repos/{o}/{r}/actions/jobs/{id}/logs`): rust-lld: error: undefined symbol: __isoc23_strtoull >>> referenced by parser.cc ... in libort_sys-....rlib rust-lld: error: undefined symbol: std::__cxx11::basic_string::_M_replace_cold(...) `ort`'s `download-binaries` feature links a prebuilt static onnxruntime (1.28.0 for ort 2.0.0-rc.13) built against glibc >= 2.38 and GCC 13's libstdc++. The runner's default job image, catthehacker/ubuntu:act-22.04, has glibc 2.35 and GCC 11. All five undefined symbols were checked in both images: act-22.04 lacks all five, act-24.04 (glibc 2.39, g++ 13) provides all five: __isoc23_strtol __isoc23_strtoll __isoc23_strtoull glibc 2.38+ basic_string::_M_replace_cold libstdc++ GCC 13+ The job now runs in catthehacker/ubuntu:act-24.04. No Rust or Cargo change, so nothing here touches the files the landing chain edits. It ran in ~1 minute because the runner's Python caches are warm; the fast failure was never early. A correction: an earlier version of this commit replaced actions/setup-python with uv, on the theory — reached by elimination, without the log — that setup-python was the fault. The log refutes that: setup-python worked in every run ("Create Python 3.11.16 folder"), and the old runs fail at the identical link error. The workflow is back on setup-python exactly as main has it. My elimination had verified the Python side in Linux containers but built the Rust side natively on macOS, where the prebuilt is a Mac binary and glibc never enters — so it cleared everything except the one step it never reproduced. 2. Encode the fixtures the way the decoder evaluates them. With the link fixed, the comparisons would have exposed a real mismatch. The Rust decoder opens every ONNX model via `ort_session::load_deterministic` — one intra-op thread, one inter-op thread, full optimisation, on purpose. The fixture generator used onnxruntime's defaults, which use every core; a convolution's reduction is split across threads and its float output shifts. For MLIC++ that is fatal. Traced symbol by symbol, encoder and decoder agree on the first 416 of 12,288 symbols — bypass-coded ones included — then diverge inside the first pass, before anything is fed back: element (ch 3, row 4, col 0) scale CDF encoder, all cores 0.78787208 16 decoder, single thread 0.78788924 17 scale_table[16] 0.78788900 One scale straddles one threshold and the arithmetic-coded stream desyncs: 11,841 symbols wrong, max|delta|=255. On main the garbage overflows the bypass loop and panics at rans.rs:77; the landing chain bounds that loop, so there it decodes silently to a wrong image. Ruled out, each by experiment: the ML stack and numpy 2 -> 1.26 downgrade (reproduced on linux/amd64); onnxruntime version drift (encoder pinned to the decoder's 1.28.0 — still broken); the bypass decode (first 64 symbols match); the hyper-latent path (decoded z bit-identical, FNV over 768 values); `build_index` (scale_table[0] is exactly 0.11). All 12 session sites now go through `_session`, mirroring load_deterministic. MLIC++ max|delta| 255 -> 0; DCVC 1 -> 0. ELIC's residual max|delta|=1 does NOT change, so it has another source (likely a final 8-bit rounding tie); it is inside the +/-4 tolerance and is not claimed as fixed. 3. A correction to this PR's own guard. The filter `_matches_reference` selects six tests, but `snac_real_model_matches_reference` needs real pretrained SNAC weights this synthetic job never builds, so it would skip on every run and keep the job red with every decoder correct. I had verified the guard fires with no fixtures, but never ran it with fixtures present. It is now excluded by name, with the reason, so the guard stays strict for the five in scope. Also corrects main()'s comment, which still described the old "exit non-zero only if ALL failed" rule. Verified on main's decoder with the job's exact commands: 5/5 fixtures (exit 0), 5 passed / 0 failed, guard clean. Falsified: removing only the threading pin brings the MLIC++ panic back; deleting one in-scope fixture makes the guard fail the step. The container change is verified by this PR's CI run. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/wai-neural-e2e.yml | 23 ++++++++++++- wai/tools/wai_validate_synthetic.py | 49 +++++++++++++++++++++------- 2 files changed, 59 insertions(+), 13 deletions(-) diff --git a/.github/workflows/wai-neural-e2e.yml b/.github/workflows/wai-neural-e2e.yml index eec8a48f2..cebedadc4 100644 --- a/.github/workflows/wai-neural-e2e.yml +++ b/.github/workflows/wai-neural-e2e.yml @@ -30,6 +30,20 @@ jobs: neural-e2e: name: synthetic fixtures → Rust decode → reference (±4) runs-on: linux + # The Rust decoders link `ort`, whose `download-binaries` feature pulls a + # prebuilt static onnxruntime (1.28.0 for ort 2.0.0-rc.13). That library was + # built against glibc >= 2.38 and GCC 13's libstdc++. The runner's default + # job image, catthehacker/ubuntu:act-22.04, has glibc 2.35 and GCC 11, so the + # test binary fails to LINK with five undefined symbols: + # __isoc23_strtol __isoc23_strtoll __isoc23_strtoull (glibc 2.38+) + # basic_string::_M_replace_cold basic_string::_M_replace_cold + # (libstdc++, GCC 13+) + # That link error is why this job failed at ~1 minute on every run: the + # Python steps pass on a warm cache and the job dies at its last step. Checked + # in both images: act-22.04 lacks all five, act-24.04 (glibc 2.39, g++ 13) + # provides all five. + container: + image: catthehacker/ubuntu:act-24.04 steps: - uses: actions/checkout@v5 - uses: actions/setup-python@v5 @@ -73,10 +87,17 @@ jobs: # succeeds while writing nothing, this job goes green having verified # nothing at all — the failure mode is silent and looks like success. # Failing on "skipping" is what makes the job's name true. + # + # One test is excluded by name, not tolerated as a skip: + # `snac_real_model_matches_reference` needs real pretrained SNAC weights, + # which this synthetic-fixture job never builds, so it would skip on every + # run and the guard would fail a job whose five in-scope decoders all + # match. Excluding it keeps the guard strict for everything in scope. run: | set -o pipefail cargo test --no-default-features --features neural --test neural_native_decode \ - -- _matches_reference --nocapture 2>&1 | tee /tmp/neural_e2e.log + -- _matches_reference --skip snac_real_model_matches_reference --nocapture \ + 2>&1 | tee /tmp/neural_e2e.log if grep -q "skipping" /tmp/neural_e2e.log; then echo "::error::a reference comparison was SKIPPED — its fixtures are missing, so this job verified nothing" grep "skipping" /tmp/neural_e2e.log diff --git a/wai/tools/wai_validate_synthetic.py b/wai/tools/wai_validate_synthetic.py index fbe356212..f71c39572 100644 --- a/wai/tools/wai_validate_synthetic.py +++ b/wai/tools/wai_validate_synthetic.py @@ -120,6 +120,31 @@ def write_wai(path, cap, kind, header, z, y, H, W): path.write_bytes(b"WAI1" + struct.pack(" int: # Each harness is independent; a failure in one must not block the others # (e.g. a synthetic model that only traces at certain sizes). Report a - # per-decoder PASS/FAIL summary and exit non-zero only if ALL failed. + # per-decoder PASS/FAIL summary, and exit non-zero unless EVERY harness succeeded. harnesses = [ ("elic", lambda: roundtrip_elic(build(MiniElic, 0), x, Path("/tmp/wai_elic"), args.size)), ("mlicpp", lambda: roundtrip_mlicpp(build(MiniMlic, 1), x, Path("/tmp/wai_mlic"), args.size)), -- 2.54.0 (Apple Git-157)