joule-moq-relay: the cascade tests wait for a refusal, not only for what got through #610

Merged

dcharlot merged 2 commits from test/relay-door-race into main 2026-11-03 22:15:42 -05:00

Conversation 0 Commits 2 Files changed 1 +22 -1

dcharlot commented 2026-11-03 18:38:42 -05:00

Owner

Copy link

receipts_are_checked_at_every_door failed once in CI on linux-arm64 (#605, job 75036): cascade.rs:406, "the edge's door refused the tampered object", left: 0, right: 1. This changes only the test. The relay does what the test claims; the test read the counter before the relay had any reason to have set it.

Cause

In session.rs, every incoming subgroup stream (one per group per track) is read in its own spawned task (uni_loop → data_stream → upstream_subgroup). A media object's door verdict (Door::object, which may wait up to the rule's hold_ms for its bundle) and the refused() call that increments stats.refused_at_door both happen in that group's task. The refused object is never fanned out, so nothing the subscriber receives comes after the refusal.

The test collected at the edge until it had G receipt objects and G-1 media groups, which the other groups' tasks are enough to supply. Then it read refused_at_door at once. Under load, group BAD's task can still be on its way to the door at that moment, and the read sees 0. Groups also reach the subscriber out of order: with the door disabled (below), the media groups arrived as [1, 4, 3, 0, 2, 5] and [0, 1, 5, 2, 3, 4].

The vote test (co_performance_votes_are_counted_once_in_chain_order) reads a counter in the same way. In floor.rs, seq 1 waits for seq 0. Seq 0's step counts it and wakes seq 1's task, and seq 1 is refused there, when it steps again (refused_inputs is incremented in that step). Meanwhile seq 0 is forwarded by its own task. The test read refused_inputs once seq 0 had reached both watchers, after a one-second quiet period. That has not failed in CI, but nothing orders the refusal before that read.

Fix

Both tests now wait for the refusal with until(STEP, …) (20 s), then still count it exactly: refused_at_door == 1 and (edge, home) refused inputs == (1, 0).

The other reads in cascade.rs were checked and left as they are:

• relay_loops, no_upstream, seek_refused and forwarded_upstream are incremented before the reply the test waits for.
• The zero-refusal reads in two_relays and floor_once come after the delivery of every object that could have been refused.
• Summaries and hop records are already waited for.
• The !media_e.contains(&BAD) absence check cannot fail spuriously.

Evidence

All runs are on the macOS build host. The load ran under taskpolicy -c <QoS>: 4 busy loops beside 4 test instances at once, each phase capped at 4 minutes.

run	tree	result
A: background QoS (2 at once)	main 2bef11f71	10/10 pass (about 50 s per run)
A: default QoS	main	400/400 pass
A: utility QoS	main	95/96 pass; 1 fail at cascade.rs:406, the CI message
B: utility QoS, the read instrumented (scratch)	main	132/132 read 1 at collection
C: utility QoS	this branch	160/160 pass

At this load, the natural rate is about 1 in 228 reads. To show the race without waiting for that rate, the window was widened with scratch delays in the relay, not committed: 300 ms before a door refusal is counted, and 1.5 s before a woken floor input is decided.

widened window	main's tests	this branch's tests
door test	8/8 fail at :406 (same message as CI)	8/8 pass
vote test	8/8 fail at :855	8/8 pass

The fixed tests still fail when the property they check is broken (scratch mutations, not committed):

• edge door disabled (door_on = false): 2/2 fail at :404, because the tampered group was forwarded;
• door refusal not counted: 2/2 fail at the new wait (:409) after 20 s;
• floor refusal not counted: 2/2 fail at the new wait (:861) after 20 s.

cargo test -p joule-moq-relay at cbdf2f203 on the build host exits 0. Every test binary passes, including cascade (12 passed).

🤖 Generated with Claude Code

`receipts_are_checked_at_every_door` failed once in CI on linux-arm64 (#605, job 75036): `cascade.rs:406`, "the edge's door refused the tampered object", **left: 0, right: 1**. This changes only the test. The relay does what the test claims; the test read the counter before the relay had any reason to have set it. ## Cause In `session.rs`, every incoming subgroup stream (one per group per track) is read in its own spawned task (`uni_loop` → `data_stream` → `upstream_subgroup`). A media object's door verdict (`Door::object`, which may wait up to the rule's `hold_ms` for its bundle) and the `refused()` call that increments `stats.refused_at_door` both happen in that group's task. The refused object is never fanned out, so nothing the subscriber receives comes after the refusal. The test collected at the edge until it had G receipt objects and G-1 media groups, which the other groups' tasks are enough to supply. Then it read `refused_at_door` at once. Under load, group BAD's task can still be on its way to the door at that moment, and the read sees 0. Groups also reach the subscriber out of order: with the door disabled (below), the media groups arrived as `[1, 4, 3, 0, 2, 5]` and `[0, 1, 5, 2, 3, 4]`. The vote test (`co_performance_votes_are_counted_once_in_chain_order`) reads a counter in the same way. In `floor.rs`, seq 1 waits for seq 0. Seq 0's step counts it and wakes seq 1's task, and seq 1 is refused there, when it steps again (`refused_inputs` is incremented in that step). Meanwhile seq 0 is forwarded by its own task. The test read `refused_inputs` once seq 0 had reached both watchers, after a one-second quiet period. That has not failed in CI, but nothing orders the refusal before that read. ## Fix Both tests now wait for the refusal with `until(STEP, …)` (20 s), then still count it exactly: `refused_at_door == 1` and `(edge, home) refused inputs == (1, 0)`. The other reads in `cascade.rs` were checked and left as they are: - `relay_loops`, `no_upstream`, `seek_refused` and `forwarded_upstream` are incremented before the reply the test waits for. - The zero-refusal reads in `two_relays` and `floor_once` come after the delivery of every object that could have been refused. - Summaries and hop records are already waited for. - The `!media_e.contains(&BAD)` absence check cannot fail spuriously. ## Evidence All runs are on the macOS build host. The load ran under `taskpolicy -c <QoS>`: 4 busy loops beside 4 test instances at once, each phase capped at 4 minutes. | run | tree | result | |---|---|---| | A: background QoS (2 at once) | main `2bef11f71` | 10/10 pass (about 50 s per run) | | A: default QoS | main | 400/400 pass | | A: utility QoS | main | 95/96 pass; **1 fail at `cascade.rs:406`, the CI message** | | B: utility QoS, the read instrumented (scratch) | main | 132/132 read 1 at collection | | C: utility QoS | this branch | **160/160 pass** | At this load, the natural rate is about 1 in 228 reads. To show the race without waiting for that rate, the window was widened with scratch delays in the relay, not committed: 300 ms before a door refusal is counted, and 1.5 s before a woken floor input is decided. | widened window | main's tests | this branch's tests | |---|---|---| | door test | 8/8 fail at `:406` (same message as CI) | 8/8 pass | | vote test | 8/8 fail at `:855` | 8/8 pass | The fixed tests still fail when the property they check is broken (scratch mutations, not committed): - edge door disabled (`door_on = false`): 2/2 fail at `:404`, because the tampered group was forwarded; - door refusal not counted: 2/2 fail at the new wait (`:409`) after 20 s; - floor refusal not counted: 2/2 fail at the new wait (`:861`) after 20 s. `cargo test -p joule-moq-relay` at `cbdf2f203` on the build host exits 0. Every test binary passes, including `cascade` (12 passed). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 1 commit 2026-11-03 18:38:42 -05:00

joule-moq-relay: the cascade tests wait for a refusal, not only for what got through ... cbdf2f2039

The door test read the edge's refused_at_door as soon as its subscriber held
every other group. Each group's door verdict is reached in that group's own
task, and nothing the subscriber is given waits for the tampered group's, so
under load the read could come first and see 0 (CI, linux-arm64). The vote
test reads the edge's refused inputs the same way, after a one-second grace.
Both now wait for the refusal, bounded, then count it exactly.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 1 commit 2026-11-03 19:37:42 -05:00

joule-moq-relay: the cascade tests count what the edge gave, not only what was read ... 1bf2f7fce3

Waiting for the refusal let a relay that counted it late and then forwarded the
refused object pass: the door test reads its subscriber only until it holds the
other groups, the vote test for a second past seq 0. Both now also count what
the edge gave its viewer, from its session summaries once the sessions end:
every group but the tampered one, and seq 0 alone.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot merged commit 36aa47c425 into main 2026-11-03 22:15:42 -05:00

dcharlot referenced this pull request from a commit 2026-11-03 22:15:42 -05:00

Merge pull request 'joule-moq-relay: the cascade tests wait for a refusal, not only for what got through' (#610) from test/relay-door-race into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!610

No description provided.