joule-moq-relay, co-performance §7.3: count each author's audience inputs in its chain's order #462

Merged

dcharlot merged 3 commits from fix/floor-input-chain-order into main 2026-10-25 05:18:18 -04:00

Conversation 0 Commits 3 Files changed 10 +601 -78

dcharlot commented 2026-10-25 04:09:39 -04:00

Owner

Copy link

What this fixes

The relay limits each audience member's inputs to one window (co-performance §7.3, Floor::events_filter / input_verdict, counted in f.sent). Each group of a member's events track rides its own stream, and the relay reads each stream in its own task, so two inputs sent back to back can reach the floor in either order. The floor counted them in the order they arrived. With a limit of one, it could forward the member's second input (seq 1, chained after seq 0) and refuse the first (seq 0). Subscribers then held seq 1 with a gap at seq 0. A verifier counting in chain order accepts seq 0 and refuses seq 1.

Found while debugging Train D: tests/floor.rs the_relay_refuses_invalid_audience_inputs failed 2 runs in 10 at load 40–70, and the diagnostic showed seq 1 forwarded and seq 0 refused. Train D (#451, now merged) hardened that test by sending seq 1 only after seq 0 arrives. That stopped the flake but did not fix the relay. This PR fixes the relay and keeps the hardened test.

The rule (co-performance §7.3, new paragraph)

A relay MUST count an author's inputs in the order of the author's chain (live-events §6.2), as a verifier does, never in the order they arrive.

• It decides an input once two things hold: it has received every record of the author's chain below the input's seq, and it has decided every input among those records.
• An input that arrives ahead of a gap is held for at most a bound the relay chooses, and decided once the gap fills.
• If the gap has not filled when the bound runs out, the input is refused input_out_of_order, and the relay counts the author's chain from the record after it.
• An input that arrives after the relay has forwarded a later input from the same author is also refused input_out_of_order, because counting it then would put it after an input that follows it in the chain.
• A relay that started after an author's chain began holds none of its earlier records, so it refuses that author's first input once the bound runs out.

§8 now lists input_out_of_order beside window_committed as a reason only the relay gives. §12 adds that an author who leaves a gap in its own chain holds up only its own inputs, and for no longer than the bound. CHANGELOG entry under v0.4.0.

Why §10, the corpus and the verifiers do not change: no verifier ever raises input_out_of_order. A verifier holds each chain in order (a record behind the head is seq_not_increasing). wai-rs (co_performance.rs, rr.order) and co_performance_verify.py (room["order"]) already count each input at its place in the merged order, which keeps each chain's order. So the reason follows the precedent of window_committed: it is named in §7.3 and §8, not in §10. §11's test requires §10 to list exactly the reference's codes, and the corpus is a corpus for verifiers. A vector for a reason only the relay gives would have nothing to check.

The relay (joule-moq-relay)

• Per-author chain state: each room's floor keeps each author's chain as received: Chain { next, ahead, waiting, forwarded }. These are the first seq it lacks, the seqs above that it has received, the inputs waiting (by digest, each with its seq and the instant its hold ends), and the highest seq of an input it has forwarded.
• When an input is decided: once next > seq and no input below it is waiting (Chain::ready). It is refused input_out_of_order if forwarded > seq, but only after the checks that don't depend on order (signature, form, power, window, type) have passed.
• Floor::events_filter is now async. An object with an input that cannot be decided yet waits, along with the rest of its stream, on a Notify. It wakes when another stream's object moves the author's chain on, or when the earliest hold in that chain runs out. This is the same pattern the door uses (door.rs hold_ms).
• Floor::events_step is the synchronous step at an explicit instant, with a Pass carrying what earlier steps decided. Unit tests drive it deterministically, with no real time.
• An input whose hold has run out is refused by whichever of its author's objects finds it, so an input stranded on a stream that has gone away is still refused and counted once. Each step first records all of the object's records in the chain, then refuses expired inputs, then decides the object's inputs: first in the object's order, then the rest in seq order. An object whose records are out of seq order is therefore still decided in chain order.
• Bounds: at most 1024 seqs kept above the first missing one, and 4096 chains per room (when full, a chain with nothing waiting is dropped). Dropping state can only stop an input from becoming ready. It never lets one through out of order.
• Config: [floor] hold_ms, 2 000 ms by default (floor::DEFAULT_HOLD_MS), set via Floor::holding. relay.example.toml shows it. FloorConfig now has a manual Default, and the struct literals in the tests gain ..FloorConfig::default(): tests/floor.rs (×2, including the hardened test's config line), tests/age_room_service.rs, joule-moq-room/tests/co_performance.rs and the config.rs unit test.
• Summary: input_out_of_order is counted in the session's summary under co_refused.

Design decisions

• Seq order, not prev-only order. Holding an input only until its immediate prev arrives is not enough. For a chain [vote, chat, vote] arriving last to first, the chat arrives (it is never held), and the second vote would then be counted before the first. The rule therefore needs every record below the input received, and no input below it still waiting.
• What "received" means: a seq the relay has received in any object of the author's own track, whatever was decided about it. A withheld record still counts as received, so the relay's own refusals never open a gap at the relay.
• When a hold runs out, the chain moves on. The relay counts from the record after the refused input. A relay that has restarted therefore loses at most one input per member (the first one after the restart) and is not stuck refusing everything that follows. A late record from the passed gap is decided normally, unless a later input has already been forwarded; then it is refused input_out_of_order.
• Forwarded inputs are never decided out of chain order. Every way an input can be forwarded requires that the inputs below it have arrived and been decided, or that their holds have run out, in which case they are refused. The refusal reasons for the order-independent checks are kept: an input is refused input_out_of_order only when ordering is the reason it is not forwarded.

Tests

• tests/floor.rs the_relay_counts_inputs_in_chain_order: the member's seq 1 is sent first, on its own stream. The test waits until the relay holds it (Floor::waiting), then sends seq 0. Seq 0 is forwarded and seq 1 is refused input_over_limit; the summary shows [("input_over_limit", 1)]. The hold is set to 60 s so no wait in the test can outlast it under load.
• tests/floor.rs the_relay_refuses_an_input_whose_gap_never_fills: seq 1 alone, with a 400 ms hold. It is refused input_out_of_order, and not before the hold runs out. A late seq 0 is then forwarded. The summary shows [("input_out_of_order", 1)].
• the_relay_refuses_invalid_audience_inputs (Train D's hardened test) is kept. Its config line gains ..FloorConfig::default(), and one comment clause now points at the new test instead of saying the limit counts in arrival order.
• Unit tests (src/floor.rs), with explicit instants:
  • inputs_are_counted_in_their_chains_order: seq 1 before seq 0; a chat between two votes, taken last to first; and an object whose records are out of seq order.
  • an_input_whose_gap_never_fills_is_refused_when_its_hold_runs_out: the hold boundary exactly; the chain counted from after the refused input; a late seq 0 refused once a later input was forwarded, and forwarded when none was; an input whose stream is gone, refused by its author's next object, with its repeat decided the same way.

Checks (build host via rb, at the PR head's tree)

Check	Command	Result
relay, all targets	cargo test --locked -p joule-moq-relay --all-targets	ok: lib 45, every integration target passes
relay build, warnings	cargo build --locked -p joule-moq-relay --all-targets	no warnings
clippy	cargo clippy --locked -p joule-moq-relay --all-targets; cd crates/joule-moq-room && cargo clippy --locked --all-targets	exit 0, no new notes (the remaining ones are older, in joule-moq, jwp, server.rs:172, session.rs:359)
dependent: room service	cd crates/joule-moq-room && cargo test --locked --all-targets	ok (lib 12, age_admission, co_performance, room_service ×3)
out-of-order test ×20 under load	cargo test … --test floor the_relay_counts_inputs_in_chain_order -- --exact looped	20/20 ok, host load 85–95
gap test ×20	… the_relay_refuses_an_input_whose_gap_never_fills -- --exact looped	20/20 ok
floor / age_* ×5	--test floor, --test age_rising, --test age_room_service, --test age_serving, --lib floor looped 5×	25/25 ok
lock files	cargo metadata --locked --offline for the workspace and for joule-moq-room	ok (no dependency change)
browser e2e co	BROWSER_BIN=<the host browser> e2e/run.sh --only co	1 of 1 runs passed, 29 of 29 checks ok; the relay refuses and counts each §7.3 reason; no page receives a refused input; no input_out_of_order appears
co-performance corpus, wai-rs	cargo test --no-default-features --features co_performance --lib co_performance:: and --test co_performance	ok (6, 5)
co-performance verifier	python3 -I live-conformance/co_performance_verify.py live-conformance/coperformance	PASS, 42 rooms, every mutation caught
repo checks (local)	check-tests-in-ci, check-workflow-concurrency, check-spec-refs, check-wai-section-refs, check-no-vanity-metrics, test-public-text-checks, check-comparative-prose, check-orphan-sources	all exit 0
public names	scripts/check-public-names.sh on the changed files, the commit messages and this body	clean

No workflow file changes, so wfsteps does not apply.

Falsifications (each committed first, then broken in the working tree, run, and restored)

1. Arrival order (Chain::ready always true): the_relay_counts_inputs_in_chain_order fails (exit 101) with seq 0 forwarded, seq 1 withheld. The relay forwarded the record with a non-zero prev (seq 1) and refused seq 0, which is the original bug. the_relay_refuses_an_input_whose_gap_never_fills and both unit tests also fail.
2. No hold ever runs out: the gap test fails (the relay refuses seq 1), and so does the unit hold test.
3. No rule against arriving behind a forwarded input: an_input_whose_gap_never_fills_is_refused_when_its_hold_runs_out fails (the late seq 0 is forwarded).
4. A waiting input below does not hold back one above it: inputs_are_counted_in_their_chains_order fails (in the object out of seq order, seq 1 is kept and seq 0 refused).

Not done

• Order-dependent rate counting (rate_exceeded) still sees non-input records as they arrive. Only inputs are held. The existing t ≤ e.t filter covers chains ordered by t; records with equal t are left as they were.
• A relay restart still loses the floor's counts, as before. The new rule only bounds what the restart costs (one input per member).

🤖 Generated with Claude Code

## What this fixes The relay limits each audience member's inputs to one window (co-performance §7.3, `Floor::events_filter` / `input_verdict`, counted in `f.sent`). Each group of a member's events track rides its own stream, and the relay reads each stream in its own task, so two inputs sent back to back can reach the floor in either order. The floor counted them in the order they arrived. With a limit of one, it could forward the member's second input (seq 1, chained after seq 0) and refuse the first (seq 0). Subscribers then held seq 1 with a gap at seq 0. A verifier counting in chain order accepts seq 0 and refuses seq 1. Found while debugging Train D: `tests/floor.rs` `the_relay_refuses_invalid_audience_inputs` failed 2 runs in 10 at load 40–70, and the diagnostic showed seq 1 forwarded and seq 0 refused. Train D (#451, now merged) hardened that test by sending seq 1 only after seq 0 arrives. That stopped the flake but did not fix the relay. This PR fixes the relay and keeps the hardened test. ## The rule (co-performance §7.3, new paragraph) A relay **MUST** count an author's inputs in the order of the author's chain (live-events §6.2), as a verifier does, never in the order they arrive. - It decides an input once two things hold: it has received every record of the author's chain below the input's `seq`, and it has decided every input among those records. - An input that arrives ahead of a gap is held for at most a bound the relay chooses, and decided once the gap fills. - If the gap has not filled when the bound runs out, the input is refused `input_out_of_order`, and the relay counts the author's chain from the record after it. - An input that arrives after the relay has forwarded a later input from the same author is also refused `input_out_of_order`, because counting it then would put it after an input that follows it in the chain. - A relay that started after an author's chain began holds none of its earlier records, so it refuses that author's first input once the bound runs out. §8 now lists `input_out_of_order` beside `window_committed` as a reason only the relay gives. §12 adds that an author who leaves a gap in its own chain holds up only its own inputs, and for no longer than the bound. CHANGELOG entry under v0.4.0. **Why §10, the corpus and the verifiers do not change:** no verifier ever raises `input_out_of_order`. A verifier holds each chain in order (a record behind the head is `seq_not_increasing`). `wai-rs` (`co_performance.rs`, `rr.order`) and `co_performance_verify.py` (`room["order"]`) already count each input at its place in the merged order, which keeps each chain's order. So the reason follows the precedent of `window_committed`: it is named in §7.3 and §8, not in §10. §11's test requires §10 to list exactly the reference's codes, and the corpus is a corpus for verifiers. A vector for a reason only the relay gives would have nothing to check. ## The relay (`joule-moq-relay`) - **Per-author chain state:** each room's floor keeps each author's chain as received: `Chain { next, ahead, waiting, forwarded }`. These are the first seq it lacks, the seqs above that it has received, the inputs waiting (by digest, each with its seq and the instant its hold ends), and the highest seq of an input it has forwarded. - **When an input is decided:** once `next > seq` and no input below it is waiting (`Chain::ready`). It is refused `input_out_of_order` if `forwarded > seq`, but only after the checks that don't depend on order (signature, form, power, window, type) have passed. - **`Floor::events_filter` is now async.** An object with an input that cannot be decided yet waits, along with the rest of its stream, on a `Notify`. It wakes when another stream's object moves the author's chain on, or when the earliest hold in that chain runs out. This is the same pattern the door uses (`door.rs` `hold_ms`). - **`Floor::events_step`** is the synchronous step at an explicit instant, with a `Pass` carrying what earlier steps decided. Unit tests drive it deterministically, with no real time. - **An input whose hold has run out** is refused by whichever of its author's objects finds it, so an input stranded on a stream that has gone away is still refused and counted once. Each step first records all of the object's records in the chain, then refuses expired inputs, then decides the object's inputs: first in the object's order, then the rest in seq order. An object whose records are out of seq order is therefore still decided in chain order. - **Bounds:** at most 1024 seqs kept above the first missing one, and 4096 chains per room (when full, a chain with nothing waiting is dropped). Dropping state can only stop an input from becoming ready. It never lets one through out of order. - **Config:** `[floor] hold_ms`, 2 000 ms by default (`floor::DEFAULT_HOLD_MS`), set via `Floor::holding`. `relay.example.toml` shows it. `FloorConfig` now has a manual `Default`, and the struct literals in the tests gain `..FloorConfig::default()`: `tests/floor.rs` (×2, including the hardened test's config line), `tests/age_room_service.rs`, `joule-moq-room/tests/co_performance.rs` and the `config.rs` unit test. - **Summary:** `input_out_of_order` is counted in the session's summary under `co_refused`. ## Design decisions - **Seq order, not prev-only order.** Holding an input only until its immediate `prev` arrives is not enough. For a chain [vote, chat, vote] arriving last to first, the chat arrives (it is never held), and the second vote would then be counted before the first. The rule therefore needs every record below the input received, and no input below it still waiting. - **What "received" means:** a seq the relay has received in any object of the author's own track, whatever was decided about it. A withheld record still counts as received, so the relay's own refusals never open a gap at the relay. - **When a hold runs out, the chain moves on.** The relay counts from the record after the refused input. A relay that has restarted therefore loses at most one input per member (the first one after the restart) and is not stuck refusing everything that follows. A late record from the passed gap is decided normally, unless a later input has already been forwarded; then it is refused `input_out_of_order`. - **Forwarded inputs are never decided out of chain order.** Every way an input can be forwarded requires that the inputs below it have arrived and been decided, or that their holds have run out, in which case they are refused. The refusal reasons for the order-independent checks are kept: an input is refused `input_out_of_order` only when ordering is the reason it is not forwarded. ## Tests - `tests/floor.rs` `the_relay_counts_inputs_in_chain_order`: the member's seq 1 is sent first, on its own stream. The test waits until the relay holds it (`Floor::waiting`), then sends seq 0. Seq 0 is forwarded and seq 1 is refused `input_over_limit`; the summary shows `[("input_over_limit", 1)]`. The hold is set to 60 s so no wait in the test can outlast it under load. - `tests/floor.rs` `the_relay_refuses_an_input_whose_gap_never_fills`: seq 1 alone, with a 400 ms hold. It is refused `input_out_of_order`, and not before the hold runs out. A late seq 0 is then forwarded. The summary shows `[("input_out_of_order", 1)]`. - `the_relay_refuses_invalid_audience_inputs` (Train D's hardened test) is kept. Its config line gains `..FloorConfig::default()`, and one comment clause now points at the new test instead of saying the limit counts in arrival order. - Unit tests (`src/floor.rs`), with explicit instants: - `inputs_are_counted_in_their_chains_order`: seq 1 before seq 0; a chat between two votes, taken last to first; and an object whose records are out of seq order. - `an_input_whose_gap_never_fills_is_refused_when_its_hold_runs_out`: the hold boundary exactly; the chain counted from after the refused input; a late seq 0 refused once a later input was forwarded, and forwarded when none was; an input whose stream is gone, refused by its author's next object, with its repeat decided the same way. ## Checks (build host via `rb`, at the PR head's tree) | Check | Command | Result | |---|---|---| | relay, all targets | `cargo test --locked -p joule-moq-relay --all-targets` | ok: lib 45, every integration target passes | | relay build, warnings | `cargo build --locked -p joule-moq-relay --all-targets` | no warnings | | clippy | `cargo clippy --locked -p joule-moq-relay --all-targets`; `cd crates/joule-moq-room && cargo clippy --locked --all-targets` | exit 0, no new notes (the remaining ones are older, in `joule-moq`, `jwp`, `server.rs:172`, `session.rs:359`) | | dependent: room service | `cd crates/joule-moq-room && cargo test --locked --all-targets` | ok (lib 12, age_admission, co_performance, room_service ×3) | | out-of-order test ×20 under load | `cargo test … --test floor the_relay_counts_inputs_in_chain_order -- --exact` looped | 20/20 ok, host load 85–95 | | gap test ×20 | `… the_relay_refuses_an_input_whose_gap_never_fills -- --exact` looped | 20/20 ok | | floor / age_* ×5 | `--test floor`, `--test age_rising`, `--test age_room_service`, `--test age_serving`, `--lib floor` looped 5× | 25/25 ok | | lock files | `cargo metadata --locked --offline` for the workspace and for joule-moq-room | ok (no dependency change) | | browser e2e `co` | `BROWSER_BIN=<the host browser> e2e/run.sh --only co` | 1 of 1 runs passed, 29 of 29 checks ok; the relay refuses and counts each §7.3 reason; no page receives a refused input; no `input_out_of_order` appears | | co-performance corpus, wai-rs | `cargo test --no-default-features --features co_performance --lib co_performance::` and `--test co_performance` | ok (6, 5) | | co-performance verifier | `python3 -I live-conformance/co_performance_verify.py live-conformance/coperformance` | PASS, 42 rooms, every mutation caught | | repo checks (local) | check-tests-in-ci, check-workflow-concurrency, check-spec-refs, check-wai-section-refs, check-no-vanity-metrics, test-public-text-checks, check-comparative-prose, check-orphan-sources | all exit 0 | | public names | `scripts/check-public-names.sh` on the changed files, the commit messages and this body | clean | No workflow file changes, so wfsteps does not apply. ## Falsifications (each committed first, then broken in the working tree, run, and restored) 1. **Arrival order** (`Chain::ready` always true): `the_relay_counts_inputs_in_chain_order` fails (exit 101) with `seq 0 forwarded, seq 1 withheld`. The relay forwarded the record with a non-zero `prev` (seq 1) and refused seq 0, which is the original bug. `the_relay_refuses_an_input_whose_gap_never_fills` and both unit tests also fail. 2. **No hold ever runs out**: the gap test fails (`the relay refuses seq 1`), and so does the unit hold test. 3. **No rule against arriving behind a forwarded input**: `an_input_whose_gap_never_fills_is_refused_when_its_hold_runs_out` fails (the late seq 0 is forwarded). 4. **A waiting input below does not hold back one above it**: `inputs_are_counted_in_their_chains_order` fails (in the object out of seq order, seq 1 is kept and seq 0 refused). ## Not done - Order-dependent rate counting (`rate_exceeded`) still sees non-input records as they arrive. Only inputs are held. The existing `t ≤ e.t` filter covers chains ordered by `t`; records with equal `t` are left as they were. - A relay restart still loses the floor's counts, as before. The new rule only bounds what the restart costs (one input per member). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 3 commits 2026-10-25 04:09:39 -04:00

joule-moq-relay: count each author's audience inputs in its chain's order, not in arrival order ... 7a188614b1

Each group of a member's events track rides a stream of its own, and the relay reads each
stream in a task of its own, so two inputs sent back to back reached the floor in either
order. The floor counted them as they came: with a limit of one, it could forward a member's
second input (seq 1) and refuse its first (seq 0), leaving subscribers with seq 1 and a gap at
seq 0, the opposite of what a verifier counting in chain order decides.

The floor now keeps each author's chain as it has received it (the first seq it lacks, the
seqs above it, and the inputs waiting) and decides an input once it holds every record of the
chain below it and no input below it waits. Floor::events_filter is async: an input ahead of a
gap waits, with the rest of its stream, until the gap fills or the floor's hold runs out
([floor] hold_ms, 2 000 by default). One whose hold runs out is refused input_out_of_order,
whichever object of its author's finds it so, and the chain is counted from the record after
it. An input that arrives after a later input of its author was forwarded is refused
input_out_of_order too. Floor::events_step is the synchronous step, tested with explicit
instants; Floor::waiting counts the inputs held.

Tests: tests/floor.rs sends a member's seq 1 first, waits until the relay holds it, then
sends seq 0: seq 0 forwarded, seq 1 refused input_over_limit; and seq 1 alone, refused
input_out_of_order no sooner than the hold, after which a late seq 0 is forwarded. Unit tests
cover a chat between two votes, an object whose records are out of seq order, the chain
counted from after a refused input, and an input whose stream is gone. The hardened test
from Train D is kept; its config gains `..FloorConfig::default()`.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

relay floor test: the payload check is the one an arrival-order relay fails first ... 0517a563c9

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai.live.coperformance §7.3: a relay counts an author's inputs in its chain's order ... 6b65ab5ffb

The relay MUST count an author's inputs in the order of the author's chain (live-events
§6.2), as a verifier does, never in the order they reach it: one chain's records can reach it
on several streams, one for each group. It decides an input once it has received every record
of the chain below it and decided every input among them; it holds an input that arrives
ahead of a gap for a bound of its own, refuses one whose gap has not filled by then as
input_out_of_order and counts the chain from the record after it; and it refuses as
input_out_of_order an input that arrives after a later input of its author was forwarded.

The reason is the relay's alone, as window_committed is (§8, now naming both): a verifier
holds each chain in order and counts each input where §9.2 step 4 meets it, so §10, the
corpus, wai-rs and co_performance_verify.py are unchanged. §12 notes that an author leaving a
gap in its own chain holds up only its own inputs, for no longer than the bound. CHANGELOG.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot merged commit 60efff5d71 into main 2026-10-25 05:18:18 -04:00

dcharlot referenced this pull request from a commit 2026-10-25 05:18:19 -04:00

Merge pull request 'joule-moq-relay, co-performance §7.3: count each author's audience inputs in its chain's order' (#462) from fix/floor-input-chain-order into main

dcharlot referenced this pull request from a commit 2026-10-25 14:30:33 -04:00

Merge origin/main (#462, #464) into train/pack-stack

dcharlot referenced this pull request 2026-10-25 23:11:13 -04:00

joule-moq-relay: relay-to-relay cascading, one live room across relays, the WAI evidence at every hop #475

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!462

No description provided.