wai-neural-e2e: make the job link and run, and fix the decoder mismatch it was hiding #101

Merged

dcharlot merged 3 commits from ci/neural-e2e-guard into main 2026-09-30 19:50:20 -04:00

Conversation 0 Commits 3 Files changed 2 +93 -14

dcharlot commented 2026-09-21 15:23:50 -04:00

Owner

Copy link

This job has never passed, so its reference comparisons have never executed in
CI. Two independent root causes, both established from evidence — the first from
the job's own log. First green run: 3374f391 — all five decoders exact.

1. The Rust test could not link on the runner

The job reaches its last step and fails there. From the job log:

rust-lld: error: undefined symbol: __isoc23_strtoull
  >>> referenced by parser.cc ... in libort_sys-....rlib
rust-lld: error: undefined symbol:
  std::__cxx11::basic_string<wchar_t>::_M_replace_cold(...)

ort's download-binaries links a prebuilt static onnxruntime (1.28.0 for ort
2.0.0-rc.13) built against glibc ≥ 2.38 and GCC 13's libstdc++. The
runner's default job image is catthehacker/ubuntu:act-22.04. All five undefined
symbols were checked in both images:

image	glibc	g++	the 5 symbols
act-22.04 (runner default)	2.35	11	all missing
act-24.04	2.39	13	all present

Fix: container: image: catthehacker/ubuntu:act-24.04 on this job. No Rust
or Cargo change, so nothing here touches the files the landing chain edits. The
job only looked like it failed early because the runner's Python caches are
warm.

Correction. An earlier revision of this PR replaced actions/setup-python
with uv, blaming setup-python by elimination because I had not yet found the
log. The log refutes that: setup-python worked in every run and the old runs
fail at the identical link error. The workflow is back on setup-python exactly
as main has it. My elimination had verified the Python side in Linux containers
but built the Rust side natively on macOS — where the prebuilt is a Mac binary
and glibc never enters — so it cleared everything except the one step it never
reproduced on the target platform.

2. Encode the fixtures the way the decoder evaluates them

With the link fixed, the comparisons expose a real mismatch. The Rust decoder
opens every ONNX model via ort_session::load_deterministic — one intra-op
thread, one inter-op thread, full optimisation, deliberately. The fixture
generator used onnxruntime's defaults — every core — so a convolution's
reduction was split across threads and its float output shifted.

Traced symbol by symbol, encoder and decoder agree on the first 416 of 12,288
MLIC++ symbols — bypass-coded ones included — then diverge inside the first
pass, before anything is fed back:

element (ch 3, row 4, col 0)	scale	CDF
encoder, all cores	0.78787208	16
decoder, single thread	0.78788924	17
scale_table[16]	0.78788900	—

One scale straddles one threshold and the arithmetic-coded stream desyncs:
11,841 symbols wrong, max|Δ|=255. On main that garbage overflows the bypass
loop and panics at rans.rs:77; the landing chain bounds that loop, so there it
decodes silently to a wrong image.

hypothesis	test	result
ML stack / numpy 2→1.26 downgrade	full run in a linux/amd64 container	fine
onnxruntime version drift	pin encoder to the decoder's 1.28.0	still broken
bypass decode	first 64 symbols	all match
hyper-latent z	FNV over all 768 values	bit-identical
build_index bound	scale_table[0]	exactly 0.11
thread count	remove only the threading pin	breaks again

All 12 session sites now go through _session, which mirrors
load_deterministic.

3. A correction to this PR's own guard

The filter _matches_reference selects six tests, but
snac_real_model_matches_reference needs real pretrained SNAC weights this
synthetic job never builds, so it skips on every run. As first written, the guard
would have kept the job red with every decoder correct — I had checked it fires
with no fixtures, but never ran it with fixtures. It is now excluded by name,
so the guard stays strict for the five in scope.

Result on the real runner (3374f391)

🚀  Start image=catthehacker/ubuntu:act-24.04
  snac   vs reference: max|Δ|=0
  glc    vs reference: max|Δ|=0
  elic   vs reference: max|Δ|=0
  mlicpp vs reference: max|Δ|=0
  dcvc   vs reference: max|Δ|=0
test result: ok. 5 passed; 0 failed; ... 16 filtered out
🏁  Job succeeded

Falsified locally: removing only the threading pin brings the MLIC++ panic back;
deleting one in-scope fixture makes the guard fail the step.

Other red checks on this PR: macOS energy backend (jcp-energy) is the known
IOReport meter defect, unrelated to this change. The linux-arm64 legs on
earlier commits were red because I broke that runner — see below — and have been
re-run on 2579c9c4. #109 was hit too.

Runner incident, my doing. While checking the glibc symbols I ran an amd64
copy of catthehacker/ubuntu:act-22.04 on the dev Mac's docker engine, which is
also the linux-arm64 runner's engine. That re-pointed the local tag at the
amd64 image, and every arm64 job failed instantly with "does not provide the
specified platform (linux/arm64)" from 02:45 to 02:47. Restored with
docker pull --platform linux/arm64, verified with the runner's own
docker create.

Correction to what I first wrote here. I originally said only this PR was
affected. That was wrong — my check used a time window and a duration filter
instead of reading logs. Recounted from the job logs themselves, 19 jobs failed
on the tag swap between 02:44:38 and 02:47:10, including three on #109
(cargo-class corpora, cross-arch byte-equality and energy receipts, all
linux-arm64). Those are infrastructure failures I caused, not defects in the
landing chain; they clear on #109's next push. I have not force-pushed #109 to
re-run them, because it is another session's branch in an ordered chain.

Interaction with the landing chain

Every link of #103–#109 edits rans.rs and neural_native_decode.rs; this PR
touches neither, so it merges independently. Once it lands, every PR blocked on
this job needs a re-run to pick up the fixed workflow.

🤖 Generated with Claude Code

This job has **never passed**, so its reference comparisons have never executed in CI. Two independent root causes, both established from evidence — the first from the job's own log. **First green run: 3374f391** — all five decoders exact. ### 1. The Rust test could not link on the runner The job reaches its last step and fails there. From the job log: ``` rust-lld: error: undefined symbol: __isoc23_strtoull >>> referenced by parser.cc ... in libort_sys-....rlib rust-lld: error: undefined symbol: std::__cxx11::basic_string<wchar_t>::_M_replace_cold(...) ``` `ort`'s `download-binaries` links a prebuilt static onnxruntime (1.28.0 for ort 2.0.0-rc.13) built against **glibc ≥ 2.38** and **GCC 13's libstdc++**. The runner's default job image is `catthehacker/ubuntu:act-22.04`. All five undefined symbols were checked in both images: | image | glibc | g++ | the 5 symbols | |---|---|---|---| | `act-22.04` (runner default) | 2.35 | 11 | **all missing** | | `act-24.04` | 2.39 | 13 | **all present** | **Fix:** `container: image: catthehacker/ubuntu:act-24.04` on this job. No Rust or Cargo change, so nothing here touches the files the landing chain edits. The job only *looked* like it failed early because the runner's Python caches are warm. > **Correction.** An earlier revision of this PR replaced `actions/setup-python` > with uv, blaming setup-python by elimination because I had not yet found the > log. The log refutes that: setup-python worked in every run and the old runs > fail at the identical link error. The workflow is back on setup-python exactly > as main has it. My elimination had verified the Python side in Linux containers > but built the Rust side natively on macOS — where the prebuilt is a Mac binary > and glibc never enters — so it cleared everything except the one step it never > reproduced on the target platform. ### 2. Encode the fixtures the way the decoder evaluates them With the link fixed, the comparisons expose a real mismatch. The Rust decoder opens every ONNX model via `ort_session::load_deterministic` — **one** intra-op thread, **one** inter-op thread, full optimisation, deliberately. The fixture generator used onnxruntime's defaults — **every core** — so a convolution's reduction was split across threads and its float output shifted. Traced symbol by symbol, encoder and decoder agree on the first **416 of 12,288** MLIC++ symbols — bypass-coded ones included — then diverge inside the first pass, before anything is fed back: | element (ch 3, row 4, col 0) | scale | CDF | |---|---|---| | encoder, all cores | 0.78787208 | **16** | | decoder, single thread | 0.78788924 | **17** | | `scale_table[16]` | 0.78788900 | — | One scale straddles one threshold and the arithmetic-coded stream desyncs: **11,841 symbols wrong, `max|Δ|=255`**. On main that garbage overflows the bypass loop and panics at `rans.rs:77`; the landing chain bounds that loop, so there it decodes silently to a wrong image. | hypothesis | test | result | |---|---|---| | ML stack / numpy 2→1.26 downgrade | full run in a linux/amd64 container | fine | | onnxruntime version drift | pin encoder to the decoder's 1.28.0 | still broken | | bypass decode | first 64 symbols | all match | | hyper-latent `z` | FNV over all 768 values | bit-identical | | `build_index` bound | `scale_table[0]` | exactly 0.11 | | **thread count** | **remove only the threading pin** | **breaks again** | All 12 session sites now go through `_session`, which mirrors `load_deterministic`. ### 3. A correction to this PR's own guard The filter `_matches_reference` selects **six** tests, but `snac_real_model_matches_reference` needs real pretrained SNAC weights this synthetic job never builds, so it skips on every run. As first written, the guard would have kept the job red with every decoder correct — I had checked it fires with no fixtures, but never ran it *with* fixtures. It is now excluded by name, so the guard stays strict for the five in scope. ### Result on the real runner (3374f391) ``` 🚀 Start image=catthehacker/ubuntu:act-24.04 snac vs reference: max|Δ|=0 glc vs reference: max|Δ|=0 elic vs reference: max|Δ|=0 mlicpp vs reference: max|Δ|=0 dcvc vs reference: max|Δ|=0 test result: ok. 5 passed; 0 failed; ... 16 filtered out 🏁 Job succeeded ``` Falsified locally: removing only the threading pin brings the MLIC++ panic back; deleting one in-scope fixture makes the guard fail the step. **Other red checks on this PR:** `macOS energy backend (jcp-energy)` is the known IOReport meter defect, unrelated to this change. The `linux-arm64` legs on earlier commits were red because I broke that runner — see below — and have been re-run on 2579c9c4. #109 was hit too. > **Runner incident, my doing.** While checking the glibc symbols I ran an amd64 > copy of `catthehacker/ubuntu:act-22.04` on the dev Mac's docker engine, which is > also the `linux-arm64` runner's engine. That re-pointed the local tag at the > amd64 image, and every arm64 job failed instantly with "does not provide the > specified platform (linux/arm64)" from 02:45 to 02:47. Restored with > `docker pull --platform linux/arm64`, verified with the runner's own > `docker create`. > > **Correction to what I first wrote here.** I originally said only this PR was > affected. That was wrong — my check used a time window and a duration filter > instead of reading logs. Recounted from the job logs themselves, 19 jobs failed > on the tag swap between 02:44:38 and 02:47:10, including **three on #109** > (`cargo-class corpora`, `cross-arch byte-equality` and `energy receipts`, all > `linux-arm64`). Those are infrastructure failures I caused, not defects in the > landing chain; they clear on #109's next push. I have not force-pushed #109 to > re-run them, because it is another session's branch in an ordered chain. ### Interaction with the landing chain Every link of #103–#109 edits `rans.rs` and `neural_native_decode.rs`; **this PR touches neither**, so it merges independently. Once it lands, every PR blocked on this job needs a re-run to pick up the fixed workflow. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 1 commit 2026-09-21 15:23:50 -04:00

ci(wai-neural-e2e): a skipped reference comparison is reported as a pass ... 738938dcac

Every `*_matches_reference` test in `neural_native_decode.rs` opens with

    if !wai.exists() || !ref.exists() || !model.exists() {
        eprintln!("skipping ... /tmp artifacts not present");
        return;
    }

so a missing fixture makes the test skip — and cargo reports a skip of this
shape as a **pass**. Verified by running the job's exact command on a machine
with no fixtures:

    test result: ok. 6 passed; 0 failed

Six green, nothing compared. The job is named "synthetic fixtures → Rust decode
→ reference (±4)" and in that state it verifies none of those things.

Today the job is red, so this is latent rather than active: the Python fixture
step must be failing outright, which is the one way this surfaces. But the
moment fixture generation succeeds while writing nothing — a harness quietly
producing 0 of 5, a path change, a partial install — the job goes fully green
having checked nothing, and the neural byte-exactness claim loses its only
end-to-end guard with no signal at all.

The fix asserts on the skip rather than enumerating fixture paths. A path list
here would be a second copy of the one in the test file, free to drift from it,
which is the same failure one level up. Grepping the run for "skipping" stays
correct whatever fixtures the harnesses grow.

Falsified: with no fixtures, `cargo test` exits 0 and the guard exits 1, naming
all six skipped comparisons.

Two things this does NOT do. It does not diagnose the current red — the forge
exposes no job logs, so the failing step cannot be identified from here; the
most likely cause is the unpinned `pip install torch torchvision compressai onnx
onnxruntime`, whose versions have moved since this workflow last ran. And it
does not pin those versions, because that would be fixing something I have not
diagnosed.

Worth noting how rarely this runs: its trigger paths (`wai/wai-rs/src/neural/**`,
the test, the validator) were last touched 2026-07-31, and the workflow itself
2026-08-26. A gate that fires a few times a year can rot between firings, and
this one was only exercised now because an unrelated stale-branch rebase
produced a push delta wide enough to match its paths.

Co-Authored-By: Claude Opus 5 <[email protected]>

dcharlot added 1 commit 2026-09-22 01:20:18 -04:00

wai: the fixture generator reports partial success as success ... e49cc554c3

The companion to the skip guard, and the reason it is not hypothetical.

`wai_validate_synthetic.py` ends with `return 0 if ok else 1`, and its harness
loop deliberately catches per-harness exceptions to keep going. So a run where
one of five harnesses succeeded exited **0**: the CI step went green, the four
decoders whose fixtures were never written then SKIPPED in the Rust suite, and a
skip of that shape is reported as a pass.

    5/5 -> exit 0    1/5 -> exit 0    0/5 -> exit 1

The job therefore had a path to fully green while having checked one decoder out
of five, with nothing in the output saying so. That is the scenario the previous
commit's guard was written against; it turns out the generator hands it over
rather than it needing an unlikely coincidence.

Now `return 0 if ok == len(harnesses) else 1`. Partial generation stays useful
interactively — the per-harness ✓/✗ lines print either way and a caller wanting
one fixture can ignore the exit code — but CI can no longer read a partial run
as a pass.

Kept alongside the grep guard rather than replacing it. They fail on different
things: this one refuses to claim fixtures that were not written, the guard
refuses a comparison that did not happen whatever the cause.

On the job's current red, now measured rather than assumed: the failing status
reports **1m7s on this branch and 1m5s on #54**, whose tree does not touch this
workflow at all. Near-identical, and far too early to have compiled the `neural`
feature set and reached the Rust step where the guard lives. Both die at the same
earlier Python step, so neither change here causes the red and neither can clear
it. Diagnosing it needs the job log, which this forge does not expose; the
unpinned `pip install torch torchvision compressai onnx onnxruntime` remains the
likely cause and is still not being fixed blind.

Co-Authored-By: Claude Opus 5 <[email protected]>

dcharlot changed title from ci(wai-neural-e2e): a skipped reference comparison is reported as a pass to ci(wai-neural-e2e): this job can go green having verified one decoder of five 2026-09-22 01:20:27 -04:00

dcharlot force-pushed ci/neural-e2e-guard from e49cc554c3 to 5bc79b1672 2026-09-26 05:10:40 -04:00 Compare

dcharlot force-pushed ci/neural-e2e-guard from 5bc79b1672 to 9eec011c48 2026-09-29 02:25:21 -04:00 Compare

dcharlot changed title from ci(wai-neural-e2e): this job can go green having verified one decoder of five to wai-neural-e2e: make the job run, and fix the decoder mismatch it was hiding 2026-09-29 02:27:54 -04:00

dcharlot force-pushed ci/neural-e2e-guard from 9eec011c48 to 3374f391de 2026-09-29 02:46:53 -04:00 Compare

dcharlot force-pushed ci/neural-e2e-guard from 3374f391de to 2579c9c42d 2026-09-29 09:46:17 -04:00 Compare

dcharlot changed title from wai-neural-e2e: make the job run, and fix the decoder mismatch it was hiding to wai-neural-e2e: make the job link and run, and fix the decoder mismatch it was hiding 2026-09-29 09:47:21 -04:00

dcharlot referenced this pull request from a commit 2026-09-30 02:25:16 -04:00

wai/research: sweep record, SyfAI input and energy-exposure notes at the chain's final state

dcharlot referenced this pull request 2026-09-30 02:25:30 -04:00

wai: research notes at the chain's final state; determinism-tiers and energy-measurement match the tree #113

dcharlot merged commit 080ecf628a into main 2026-09-30 19:50:20 -04:00

dcharlot referenced this pull request from a commit 2026-09-30 19:50:22 -04:00

Merge pull request 'wai-neural-e2e: make the job link and run, and fix the decoder mismatch it was hiding' (#101) from ci/neural-e2e-guard into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!101

No description provided.