ci(wai-neural-e2e): this job can go green having verified one decoder of five #101

Open

dcharlot wants to merge 2 commits from ci/neural-e2e-guard into main

pull from: ci/neural-e2e-guard

merge into: Transaction-Science:main

Transaction-Science:main

Transaction-Science:ci/macos-energy-widen

Transaction-Science:ci/smart-byte-pack

Transaction-Science:ci/tier2-packs

Transaction-Science:wai-webcodecs-negotiation

Transaction-Science:ci/macos-energy-backend

Transaction-Science:feat/wai-video-workflow

Transaction-Science:ci/joule-code-pack

Transaction-Science:ci/jcp-pack

Transaction-Science:joulecontract/run-negative-vectors

Transaction-Science:ci/jouleclaw-arm64-debuginfo

Transaction-Science:ci/run-conformance-verifiers

Transaction-Science:mesh/cost-calibration

Transaction-Science:diagnose/grounding-backing

Transaction-Science:wai/energy-operating-point

Transaction-Science:wai/2030-transport-landscape

Transaction-Science:wai/constraints-and-unidirectional

Transaction-Science:wai/determinism-tier-covers-emitted-medium

Transaction-Science:wai/moq-streaming-format

Transaction-Science:wai/mpeg-ai-part6-mapping

Transaction-Science:wai/prior-carriage-and-derivations

Transaction-Science:wai/receipt-classes-and-key-discovery

Transaction-Science:wai/task-fidelity

Transaction-Science:wai/transparency-emission

Transaction-Science:wai/c2pa-emitter

Transaction-Science:compliance/eu-scale-pack

Transaction-Science:compliance/composite-grade-v2

Transaction-Science:jcp/erasure-tombstone-v2

Transaction-Science:jcp/avoided-energy

Transaction-Science:jcp/erasure-tombstone

Transaction-Science:jcp/environment-binding

Transaction-Science:proof/stub-on-the-wire

Transaction-Science:compliance/composite-grade

Transaction-Science:mesh/context-cost-profile

Transaction-Science:ar-1-on-main

Transaction-Science:jcp/mcp-meta-receipt

Transaction-Science:wai/energy-binding

Transaction-Science:jcp/energy-coverage-relanded

Transaction-Science:jcp/energy-coverage

Transaction-Science:wai/video-byte-equality

Transaction-Science:eoc/multi-tenant-allocation

Transaction-Science:compliance/banding-function

Transaction-Science:ci-standards-workspaces

Transaction-Science:sandbox/honest-tier

Transaction-Science:proof/joule-ceiling

Transaction-Science:chore/rust-1.98-and-deps

Transaction-Science:wai-jpegai-full-wasm

Transaction-Science:wai-3d-landscape

Transaction-Science:wai-binding-real-decode

Transaction-Science:wai-determinism-tiers

Transaction-Science:wai-jpegai-wasm-demo

Transaction-Science:wai-jpegai-integrate

Transaction-Science:fix-byte-exact-includes

Transaction-Science:ci-windows-shell

Transaction-Science:jpegai-dequant-derive

Transaction-Science:ci-rust-setup

Transaction-Science:fl2-land

Transaction-Science:ci/cross-workspace-check

Transaction-Science:ci-enable

Transaction-Science:jcp-training-receipt

Transaction-Science:joule-code/repair-grant-literals

Transaction-Science:jcp-gateway-flow-relay

Transaction-Science:jcp-receipt-flow-seal

Transaction-Science:jcp-runtime-flow-gate

Transaction-Science:jcp-flow-witness

Transaction-Science:jcp-flow-lattice

Transaction-Science:rust-toolchain-1.97.1

Transaction-Science:score-binary-on-320fa1e

Transaction-Science:sandbox/cred-injection-2

Transaction-Science:sandbox/credential-injection

Transaction-Science:corpus-embedder-s5

Transaction-Science:jcp/cite

Transaction-Science:receipts-corpus-connector

Transaction-Science:jcp/grant-caveats

Transaction-Science:feat/jpegai-derive-inputs

Transaction-Science:feat/jpegai-weight-tools

Transaction-Science:feat/jpegai-icci-validation

Transaction-Science:jcp/x402-example-settlement

Transaction-Science:jcp/x402-bridge

Transaction-Science:jcp/receipt-payload-hash

Transaction-Science:fix/jpegai-pih-flags

Transaction-Science:feat/jpegai-e2e-chain

Transaction-Science:feat/jpegai-e2e-z

Transaction-Science:feat/jpegai-ton

Transaction-Science:jcp/dual-ceilings

Transaction-Science:feat/jpegai-pih

Transaction-Science:feat/jpegai-bitstream

Transaction-Science:feat/jpegai-icci-net2

Transaction-Science:feat/jpegai-icci-net

Transaction-Science:feat/jpegai-dwt

Transaction-Science:feat/jpegai-color

Transaction-Science:score-binary-rerank

Transaction-Science:feat/jpegai-efl

Transaction-Science:feat/jpegai-lef

Transaction-Science:feat/jpegai-efn

Transaction-Science:feat/jpegai-lsbs

Transaction-Science:feat/jpegai-reconstruct

Transaction-Science:jcp/orchestrate-profile-mapping

Transaction-Science:jpegai-sigma-index

Transaction-Science:hyperscale-floorfix

Transaction-Science:feat/jpegai-pipeline-dequant

Transaction-Science:feat/jpegai-e2e-synthesis

Transaction-Science:recovery/jpegai-decoders

Transaction-Science:wai/jpegai-synthesis-executor

Transaction-Science:jouletag-standard

Transaction-Science:wai/jpegai-gate-fixes

Transaction-Science:wai/int-transformer-executors

Transaction-Science:wai/int-transformer-kernels

Transaction-Science:energy/rapl-nvml-hardware-fixes

Transaction-Science:fix/omni-build-and-gguf-tokenizer

Transaction-Science:jouletable-standard

Transaction-Science:jouleclaw/gguf-decoders

Transaction-Science:wai/confidentiality-freshness

Transaction-Science:joulehook-standard

Transaction-Science:honest-counter-provenance

Transaction-Science:amd-kds-adapter

Transaction-Science:evidence-derived-grounding

Transaction-Science:mapping-physically-rooted-clarify

Transaction-Science:attested-efficiency-demo

Transaction-Science:efficiency-energy-attestation

Transaction-Science:mapping-attested-kernel

Transaction-Science:energy-counter-attestation

Transaction-Science:efficiency-surface

Transaction-Science:efficiency-spec

Transaction-Science:harness-econ

Transaction-Science:federation-quic-on-main

Conversation 0 Commits 2 Files changed 2 +35 -2

dcharlot commented 2026-09-21 15:23:50 -04:00

Owner

Copy link

Two commits, one defect in two halves: this job has a path to fully green
while verifying almost nothing.

Half 1 — a skipped comparison is reported as a pass

Every *_matches_reference test in neural_native_decode.rs opens with:

if !wai.exists() || !ref.exists() || !model.exists() {
    eprintln!("skipping ... /tmp artifacts not present");
    return;
}

Run the job's exact command with no fixtures:

test result: ok. 6 passed; 0 failed

Six green, nothing compared, in a job named "synthetic fixtures → Rust decode →
reference (±4)". Not ignored, not filtered out — passed.

Half 2 — the generator hands that scenario over

I first wrote that half 1 needed an unlikely coincidence to bite. It does not.
wai_validate_synthetic.py ended with return 0 if ok else 1, and its harness
loop deliberately catches per-harness exceptions to keep going:

harnesses succeeded	exit	CI step
5 / 5	0	passes
1 / 5	0	passes
0 / 5	1	fails

So one working harness was enough for a green fixture step — after which the
four decoders whose fixtures were never written skip, report as passes, and the
job comes out fully green having checked one decoder out of five, with
nothing in the output saying so.

Now return 0 if ok == len(harnesses) else 1. Partial generation stays useful
interactively (the per-harness ✓/✗ lines print either way; a caller wanting one
fixture can ignore the exit code), but CI can no longer read a partial run as a
pass.

Both kept, because they fail on different things

• The generator refuses to claim fixtures it did not write.
• The guard refuses a comparison that did not happen, whatever the cause.

The guard asserts on the skip rather than on a fixture path list, because a path
list in the workflow would be a second copy of the one in the test file, free to
drift from it — the same bug one level up.

Falsified

With no fixtures present: cargo test exits 0, the guard exits 1, naming
all six skipped comparisons.

On this PR's own red — measured, not assumed

The failing status reports 1m7s on this branch and 1m5s on #54, whose
tree does not touch this workflow at all. Near-identical, and far too early to
have compiled the neural feature set and reached the Rust step where the guard
lives. Both die at the same earlier Python step.

So neither change here causes the red, and neither can clear it. Diagnosing
it needs the job log, which this forge does not expose. The unpinned
pip install torch torchvision compressai onnx onnxruntime remains the likely
cause — its versions have moved since this workflow last ran — and I am not
pinning it blind.

How rarely this gate fires

Trigger paths (wai/wai-rs/src/neural/**, the test, the validator) last touched
2026-07-31; the workflow itself 2026-08-26. A gate that fires a few times
a year rots between firings — and it was only exercised now because an unrelated
stale-branch rebase produced a push delta wide enough to match its paths.

🤖 Generated with Claude Code

Two commits, one defect in two halves: **this job has a path to fully green while verifying almost nothing.** ### Half 1 — a skipped comparison is reported as a pass Every `*_matches_reference` test in `neural_native_decode.rs` opens with: ```rust if !wai.exists() || !ref.exists() || !model.exists() { eprintln!("skipping ... /tmp artifacts not present"); return; } ``` Run the job's exact command with no fixtures: ``` test result: ok. 6 passed; 0 failed ``` Six green, nothing compared, in a job named *"synthetic fixtures → Rust decode → reference (±4)"*. Not `ignored`, not `filtered out` — **passed**. ### Half 2 — the generator hands that scenario over I first wrote that half 1 needed an unlikely coincidence to bite. It does not. `wai_validate_synthetic.py` ended with `return 0 if ok else 1`, and its harness loop deliberately catches per-harness exceptions to keep going: | harnesses succeeded | exit | CI step | |---|---|---| | 5 / 5 | 0 | passes | | **1 / 5** | **0** | **passes** | | 0 / 5 | 1 | fails | So one working harness was enough for a green fixture step — after which the four decoders whose fixtures were never written skip, report as passes, and the job comes out **fully green having checked one decoder out of five**, with nothing in the output saying so. Now `return 0 if ok == len(harnesses) else 1`. Partial generation stays useful interactively (the per-harness ✓/✗ lines print either way; a caller wanting one fixture can ignore the exit code), but CI can no longer read a partial run as a pass. ### Both kept, because they fail on different things - The **generator** refuses to claim fixtures it did not write. - The **guard** refuses a comparison that did not happen, whatever the cause. The guard asserts on the skip rather than on a fixture path list, because a path list in the workflow would be a second copy of the one in the test file, free to drift from it — the same bug one level up. ### Falsified With no fixtures present: `cargo test` exits **0**, the guard exits **1**, naming all six skipped comparisons. ### On this PR's own red — measured, not assumed The failing status reports **1m7s on this branch** and **1m5s on #54**, whose tree does not touch this workflow at all. Near-identical, and far too early to have compiled the `neural` feature set and reached the Rust step where the guard lives. Both die at the same earlier Python step. **So neither change here causes the red, and neither can clear it.** Diagnosing it needs the job log, which this forge does not expose. The unpinned `pip install torch torchvision compressai onnx onnxruntime` remains the likely cause — its versions have moved since this workflow last ran — and I am not pinning it blind. ### How rarely this gate fires Trigger paths (`wai/wai-rs/src/neural/**`, the test, the validator) last touched **2026-07-31**; the workflow itself **2026-08-26**. A gate that fires a few times a year rots between firings — and it was only exercised now because an unrelated stale-branch rebase produced a push delta wide enough to match its paths. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 1 commit 2026-09-21 15:23:50 -04:00

ci(wai-neural-e2e): a skipped reference comparison is reported as a pass ... 738938dcac

Every `*_matches_reference` test in `neural_native_decode.rs` opens with

    if !wai.exists() || !ref.exists() || !model.exists() {
        eprintln!("skipping ... /tmp artifacts not present");
        return;
    }

so a missing fixture makes the test skip — and cargo reports a skip of this
shape as a **pass**. Verified by running the job's exact command on a machine
with no fixtures:

    test result: ok. 6 passed; 0 failed

Six green, nothing compared. The job is named "synthetic fixtures → Rust decode
→ reference (±4)" and in that state it verifies none of those things.

Today the job is red, so this is latent rather than active: the Python fixture
step must be failing outright, which is the one way this surfaces. But the
moment fixture generation succeeds while writing nothing — a harness quietly
producing 0 of 5, a path change, a partial install — the job goes fully green
having checked nothing, and the neural byte-exactness claim loses its only
end-to-end guard with no signal at all.

The fix asserts on the skip rather than enumerating fixture paths. A path list
here would be a second copy of the one in the test file, free to drift from it,
which is the same failure one level up. Grepping the run for "skipping" stays
correct whatever fixtures the harnesses grow.

Falsified: with no fixtures, `cargo test` exits 0 and the guard exits 1, naming
all six skipped comparisons.

Two things this does NOT do. It does not diagnose the current red — the forge
exposes no job logs, so the failing step cannot be identified from here; the
most likely cause is the unpinned `pip install torch torchvision compressai onnx
onnxruntime`, whose versions have moved since this workflow last ran. And it
does not pin those versions, because that would be fixing something I have not
diagnosed.

Worth noting how rarely this runs: its trigger paths (`wai/wai-rs/src/neural/**`,
the test, the validator) were last touched 2026-07-31, and the workflow itself
2026-08-26. A gate that fires a few times a year can rot between firings, and
this one was only exercised now because an unrelated stale-branch rebase
produced a push delta wide enough to match its paths.

Co-Authored-By: Claude Opus 5 <[email protected]>

dcharlot added 1 commit 2026-09-22 01:20:18 -04:00

wai: the fixture generator reports partial success as success ... e49cc554c3

The companion to the skip guard, and the reason it is not hypothetical.

`wai_validate_synthetic.py` ends with `return 0 if ok else 1`, and its harness
loop deliberately catches per-harness exceptions to keep going. So a run where
one of five harnesses succeeded exited **0**: the CI step went green, the four
decoders whose fixtures were never written then SKIPPED in the Rust suite, and a
skip of that shape is reported as a pass.

    5/5 -> exit 0    1/5 -> exit 0    0/5 -> exit 1

The job therefore had a path to fully green while having checked one decoder out
of five, with nothing in the output saying so. That is the scenario the previous
commit's guard was written against; it turns out the generator hands it over
rather than it needing an unlikely coincidence.

Now `return 0 if ok == len(harnesses) else 1`. Partial generation stays useful
interactively — the per-harness ✓/✗ lines print either way and a caller wanting
one fixture can ignore the exit code — but CI can no longer read a partial run
as a pass.

Kept alongside the grep guard rather than replacing it. They fail on different
things: this one refuses to claim fixtures that were not written, the guard
refuses a comparison that did not happen whatever the cause.

On the job's current red, now measured rather than assumed: the failing status
reports **1m7s on this branch and 1m5s on #54**, whose tree does not touch this
workflow at all. Near-identical, and far too early to have compiled the `neural`
feature set and reached the Rust step where the guard lives. Both die at the same
earlier Python step, so neither change here causes the red and neither can clear
it. Diagnosing it needs the job log, which this forge does not expose; the
unpinned `pip install torch torchvision compressai onnx onnxruntime` remains the
likely cause and is still not being fixed blind.

Co-Authored-By: Claude Opus 5 <[email protected]>

dcharlot changed title from ci(wai-neural-e2e): a skipped reference comparison is reported as a pass to ci(wai-neural-e2e): this job can go green having verified one decoder of five 2026-09-22 01:20:27 -04:00

dcharlot force-pushed ci/neural-e2e-guard from e49cc554c3 to 5bc79b1672 2026-09-26 05:10:40 -04:00 Compare

Some checks failed

Hide checks

standards workspaces build + test / jouleclaw-rs (pull_request) Successful in 2m8s

Details

standards workspaces build + test / aac-rs (pull_request) Successful in 34s

Details

standards workspaces build + test / joule-code-rs (pull_request) Successful in 35s

Details

standards workspaces build + test / joulecontract-rs (pull_request) Successful in 38s

Details

standards workspaces build + test / map-rs (pull_request) Successful in 33s

Details

standards workspaces build + test / mesh-rs (pull_request) Successful in 27s

Details

standards workspaces build + test / eoc-rs (pull_request) Successful in 1m46s

Details

standards workspaces build + test / jcp-rs (pull_request) Successful in 1m17s

Details

standards workspaces build + test / openpay (pull_request) Successful in 1m34s

Details

standards workspaces build + test / proof-rs (pull_request) Successful in 1m16s

Details

standards workspaces build + test / sandbox-rs (pull_request) Successful in 2m8s

Details

standards workspaces build + test / smart-byte-rs (pull_request) Successful in 2m35s

Details

WAI neural decoders end-to-end / synthetic fixtures → Rust decode → reference (±4) (pull_request) Failing after 1m6s

Details

standards workspaces build + test / arl-rs (pull_request) Successful in 3m33s

Details

standards workspaces build + test / joule-ui-rs (pull_request) Successful in 2m11s

Details

standards workspaces build + test / jouledesk-rs (pull_request) Successful in 2m9s

Details

standards workspaces build + test / joulehook-rs (pull_request) Successful in 2m17s

Details

standards workspaces build + test / sense-rs (pull_request) Successful in 2m12s

Details

standards workspaces build + test / macOS energy backend (jcp-energy) (pull_request) Failing after 2m11s

Details

This pull request can be merged automatically.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin ci/neural-e2e-guard:ci/neural-e2e-guard

git switch ci/neural-e2e-guard

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff ci/neural-e2e-guard

git switch ci/neural-e2e-guard

git rebase main

git switch main

git merge --ff-only ci/neural-e2e-guard

git switch ci/neural-e2e-guard

git rebase main

git switch main

git merge --no-ff ci/neural-e2e-guard

git switch main

git merge --squash ci/neural-e2e-guard

git switch main

git merge --ff-only ci/neural-e2e-guard

git switch main

git merge ci/neural-e2e-guard

git push origin main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!101

No description provided.