wai-neural-e2e: make the job link and run, and fix the decoder mismatch it was hiding #101

Merged

dcharlot merged 3 commits from ci/neural-e2e-guard into main 2026-09-30 19:50:20 -04:00

Conversation 0 Commits 3 Files changed 2 +93 -14

2026-09-29

Transaction Science

wai-neural-e2e: make the job run, and fix the decoder mismatch it was hiding ...

This job has failed at ~1 minute on every run, so its reference comparisons have
never executed in CI. Two root causes, both now established from evidence — the
first from the job's actual log.

1. The Rust test could not link on the runner.

The job reaches its last step and fails there. From the job log (forge 16.0.5
exposes `GET /repos/{o}/{r}/actions/jobs/{id}/logs`):

    rust-lld: error: undefined symbol: __isoc23_strtoull
      >>> referenced by parser.cc ... in libort_sys-....rlib
    rust-lld: error: undefined symbol:
      std::__cxx11::basic_string<wchar_t>::_M_replace_cold(...)

`ort`'s `download-binaries` feature links a prebuilt static onnxruntime (1.28.0
for ort 2.0.0-rc.13) built against glibc >= 2.38 and GCC 13's libstdc++. The
runner's default job image, catthehacker/ubuntu:act-22.04, has glibc 2.35 and
GCC 11. All five undefined symbols were checked in both images: act-22.04 lacks
all five, act-24.04 (glibc 2.39, g++ 13) provides all five:

    __isoc23_strtol  __isoc23_strtoll  __isoc23_strtoull      glibc 2.38+
    basic_string<char|wchar_t>::_M_replace_cold               libstdc++ GCC 13+

The job now runs in catthehacker/ubuntu:act-24.04. No Rust or Cargo change, so
nothing here touches the files the landing chain edits. It ran in ~1 minute
because the runner's Python caches are warm; the fast failure was never early.

A correction: an earlier version of this commit replaced actions/setup-python
with uv, on the theory — reached by elimination, without the log — that
setup-python was the fault. The log refutes that: setup-python worked in every
run ("Create Python 3.11.16 folder"), and the old runs fail at the identical
link error. The workflow is back on setup-python exactly as main has it. My
elimination had verified the Python side in Linux containers but built the Rust
side natively on macOS, where the prebuilt is a Mac binary and glibc never
enters — so it cleared everything except the one step it never reproduced.

2. Encode the fixtures the way the decoder evaluates them.

With the link fixed, the comparisons would have exposed a real mismatch. The
Rust decoder opens every ONNX model via `ort_session::load_deterministic` — one
intra-op thread, one inter-op thread, full optimisation, on purpose. The fixture
generator used onnxruntime's defaults, which use every core; a convolution's
reduction is split across threads and its float output shifts.

For MLIC++ that is fatal. Traced symbol by symbol, encoder and decoder agree on
the first 416 of 12,288 symbols — bypass-coded ones included — then diverge
inside the first pass, before anything is fed back:

    element (ch 3, row 4, col 0)     scale         CDF
    encoder, all cores               0.78787208    16
    decoder, single thread           0.78788924    17
    scale_table[16]                  0.78788900

One scale straddles one threshold and the arithmetic-coded stream desyncs:
11,841 symbols wrong, max|delta|=255. On main the garbage overflows the bypass
loop and panics at rans.rs:77; the landing chain bounds that loop, so there it
decodes silently to a wrong image.

Ruled out, each by experiment: the ML stack and numpy 2 -> 1.26 downgrade
(reproduced on linux/amd64); onnxruntime version drift (encoder pinned to the
decoder's 1.28.0 — still broken); the bypass decode (first 64 symbols match);
the hyper-latent path (decoded z bit-identical, FNV over 768 values);
`build_index` (scale_table[0] is exactly 0.11).

All 12 session sites now go through `_session`, mirroring load_deterministic.
MLIC++ max|delta| 255 -> 0; DCVC 1 -> 0. ELIC's residual max|delta|=1 does NOT
change, so it has another source (likely a final 8-bit rounding tie); it is
inside the +/-4 tolerance and is not claimed as fixed.

3. A correction to this PR's own guard.

The filter `_matches_reference` selects six tests, but
`snac_real_model_matches_reference` needs real pretrained SNAC weights this
synthetic job never builds, so it would skip on every run and keep the job red
with every decoder correct. I had verified the guard fires with no fixtures, but
never ran it with fixtures present. It is now excluded by name, with the reason,
so the guard stays strict for the five in scope.

Also corrects main()'s comment, which still described the old
"exit non-zero only if ALL failed" rule.

Verified on main's decoder with the job's exact commands: 5/5 fixtures (exit 0),
5 passed / 0 failed, guard clean. Falsified: removing only the threading pin
brings the MLIC++ panic back; deleting one in-scope fixture makes the guard fail
the step. The container change is verified by this PR's CI run.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

2579c9c42d

• View at this point in history

2026-09-29 09:46:15 -04:00

Transaction Science

wai: the fixture generator reports partial success as success ...

The companion to the skip guard, and the reason it is not hypothetical.

`wai_validate_synthetic.py` ends with `return 0 if ok else 1`, and its harness
loop deliberately catches per-harness exceptions to keep going. So a run where
one of five harnesses succeeded exited **0**: the CI step went green, the four
decoders whose fixtures were never written then SKIPPED in the Rust suite, and a
skip of that shape is reported as a pass.

    5/5 -> exit 0    1/5 -> exit 0    0/5 -> exit 1

The job therefore had a path to fully green while having checked one decoder out
of five, with nothing in the output saying so. That is the scenario the previous
commit's guard was written against; it turns out the generator hands it over
rather than it needing an unlikely coincidence.

Now `return 0 if ok == len(harnesses) else 1`. Partial generation stays useful
interactively — the per-harness ✓/✗ lines print either way and a caller wanting
one fixture can ignore the exit code — but CI can no longer read a partial run
as a pass.

Kept alongside the grep guard rather than replacing it. They fail on different
things: this one refuses to claim fixtures that were not written, the guard
refuses a comparison that did not happen whatever the cause.

On the job's current red, now measured rather than assumed: the failing status
reports **1m7s on this branch and 1m5s on #54**, whose tree does not touch this
workflow at all. Near-identical, and far too early to have compiled the `neural`
feature set and reached the Rust step where the guard lives. Both die at the same
earlier Python step, so neither change here causes the red and neither can clear
it. Diagnosing it needs the job log, which this forge does not expose; the
unpinned `pip install torch torchvision compressai onnx onnxruntime` remains the
likely cause and is still not being fixed blind.

Co-Authored-By: Claude Opus 5 <[email protected]>

7508665026

• View at this point in history

2026-09-29 02:20:08 -04:00

Transaction Science

ci(wai-neural-e2e): a skipped reference comparison is reported as a pass ...

Every `*_matches_reference` test in `neural_native_decode.rs` opens with

    if !wai.exists() || !ref.exists() || !model.exists() {
        eprintln!("skipping ... /tmp artifacts not present");
        return;
    }

so a missing fixture makes the test skip — and cargo reports a skip of this
shape as a **pass**. Verified by running the job's exact command on a machine
with no fixtures:

    test result: ok. 6 passed; 0 failed

Six green, nothing compared. The job is named "synthetic fixtures → Rust decode
→ reference (±4)" and in that state it verifies none of those things.

Today the job is red, so this is latent rather than active: the Python fixture
step must be failing outright, which is the one way this surfaces. But the
moment fixture generation succeeds while writing nothing — a harness quietly
producing 0 of 5, a path change, a partial install — the job goes fully green
having checked nothing, and the neural byte-exactness claim loses its only
end-to-end guard with no signal at all.

The fix asserts on the skip rather than enumerating fixture paths. A path list
here would be a second copy of the one in the test file, free to drift from it,
which is the same failure one level up. Grepping the run for "skipping" stays
correct whatever fixtures the harnesses grow.

Falsified: with no fixtures, `cargo test` exits 0 and the guard exits 1, naming
all six skipped comparisons.

Two things this does NOT do. It does not diagnose the current red — the forge
exposes no job logs, so the failing step cannot be identified from here; the
most likely cause is the unpinned `pip install torch torchvision compressai onnx
onnxruntime`, whose versions have moved since this workflow last ran. And it
does not pin those versions, because that would be fixing something I have not
diagnosed.

Worth noting how rarely this runs: its trigger paths (`wai/wai-rs/src/neural/**`,
the test, the validator) were last touched 2026-07-31, and the workflow itself
2026-08-26. A gate that fires a few times a year can rot between firings, and
this one was only exercised now because an unrelated stale-branch rebase
produced a push delta wide enough to match its paths.

Co-Authored-By: Claude Opus 5 <[email protected]>

f82d81e5e9

• View at this point in history

2026-09-29 02:20:08 -04:00