sandbox-meter: a member waiting on the kernel in the middle of its work is not settled #230

Merged

dcharlot merged 2 commits from fix/tree-energy-reap-under-load into main 2026-10-13 03:13:53 -04:00

Conversation 0 Commits 2 Files changed 5 +346 -50

dcharlot commented 2026-10-13 01:03:53 -04:00

Owner

Copy link

What

Two sandbox-meter process-tree tests failed on loaded hosts. Looking into them found a defect in the tree meter's settle rule, which this fixes, and one test whose claim was stronger than the meter's contract.

The defect

The meter took a member for settled when two reads of its counts agreed with no thread on a core or queued between them. A thread in a wait that no signal ends (the kernel's own waits: a block being written, a page being brought in, a lock) passes that rule, because it is off its core and folded. But it runs again when the kernel is done, with nothing arriving, so the member is in the middle of its work. A stop taken there read the span short by the rest of that work, and the reading was whole, with no gap.

Measured on an M4 Max, macOS 27.0.1, with a server whose reply is followed by twenty rounds of a 4 KiB write, an fsync and a short burn:

Rule	Round trips	Short	Exact
run-state count alone	60	24 (at 16 to 92% of the oracle's advance, each whole)	36
with each thread's wait state	60	0	60

The rule

Once pti_numrunning is zero, the meter asks each thread's own run state (PROC_PIDLISTTHREADS, then PROC_PIDTHREADINFO). A member with a thread in TH_STATE_UNINTERRUPTIBLE is not settled: it is walked again, and is TreeGap::Unsettled if it never comes to rest inside the bound.

• A server waiting for its next request settles as before. A thread asleep, or blocked reading a pipe, was in that state at 0 of 20 looks each (a thread in an fsync loop: 18 of 60), and none of an idle host's threads was at 5 looks.
• Cost: the start of a span on a tree of 61 processes took 1.73 to 1.75 ms against 1.68 to 1.72 ms (least of 300 starts, three runs each).
• JCP SPEC §5.1 now states the requirement, and names the second cause beside "still running" in the gap it already has.

What it does not close: a member waiting mid-work on something a signal could end (a pipe, a socket, a child, a timer) still reads as settled. The module docs say so.

The tests

• a_worker_reaped_while_its_tree_is_walked_is_never_missed asserted that no reading is whole. In one CI run, one round trip of 21 read the worker alive and off every core, counted its record, and was whole: the worker was read, not missed. The test now asserts its name: a whole reading must hold the worker as a member.
• a_server_that_reaps_its_worker_is_not_whole failed its equality with the oracle once, at a load past 100 on a host short of memory. That is what a member read while it waits for a page looks like. The run's figures were not kept, so this is the likely cause, not a demonstrated one; the test is unchanged.
• New: a_reply_whose_tail_waits_on_the_disk_is_read_once_the_tail_is_done. It fails against the earlier rule on an idle host and passes with this one.
• The oracle's own quiet rule now asks the same of each thread, read raw, so it is not fooled where the meter no longer is.

Verification

On the build Mac:

• cargo test -p sandbox-meter: all pass idle, and in each of twelve full runs under a load past 160 (72 disk-wait round trips, all exact, none with a gap).
• The process-tree tests of the crates that use this meter all pass: jcp-gateway, proof-solvers, sandbox-proof, joule-code-remediate.

Neither original failure reproduced under CPU load alone, before or after the change; both were seen only on a host that was also short of memory.

🤖 Generated with Claude Code

## What Two `sandbox-meter` process-tree tests failed on loaded hosts. Looking into them found a defect in the tree meter's settle rule, which this fixes, and one test whose claim was stronger than the meter's contract. ## The defect The meter took a member for settled when two reads of its counts agreed with no thread on a core or queued between them. A thread in a wait that no signal ends (the kernel's own waits: a block being written, a page being brought in, a lock) passes that rule, because it is off its core and folded. But it runs again when the kernel is done, with nothing arriving, so the member is in the middle of its work. A stop taken there read the span short by the rest of that work, and the reading was whole, with no gap. Measured on an M4 Max, macOS 27.0.1, with a server whose reply is followed by twenty rounds of a 4 KiB write, an `fsync` and a short burn: | Rule | Round trips | Short | Exact | |---|---|---|---| | run-state count alone | 60 | 24 (at 16 to 92% of the oracle's advance, each whole) | 36 | | with each thread's wait state | 60 | 0 | 60 | ## The rule Once `pti_numrunning` is zero, the meter asks each thread's own run state (`PROC_PIDLISTTHREADS`, then `PROC_PIDTHREADINFO`). A member with a thread in `TH_STATE_UNINTERRUPTIBLE` is not settled: it is walked again, and is `TreeGap::Unsettled` if it never comes to rest inside the bound. - A server waiting for its next request settles as before. A thread asleep, or blocked reading a pipe, was in that state at 0 of 20 looks each (a thread in an `fsync` loop: 18 of 60), and none of an idle host's threads was at 5 looks. - Cost: the start of a span on a tree of 61 processes took 1.73 to 1.75 ms against 1.68 to 1.72 ms (least of 300 starts, three runs each). - JCP SPEC §5.1 now states the requirement, and names the second cause beside "still running" in the gap it already has. What it does not close: a member waiting mid-work on something a signal could end (a pipe, a socket, a child, a timer) still reads as settled. The module docs say so. ## The tests - `a_worker_reaped_while_its_tree_is_walked_is_never_missed` asserted that no reading is whole. In one CI run, one round trip of 21 read the worker alive and off every core, counted its record, and was whole: the worker was read, not missed. The test now asserts its name: a whole reading must hold the worker as a member. - `a_server_that_reaps_its_worker_is_not_whole` failed its equality with the oracle once, at a load past 100 on a host short of memory. That is what a member read while it waits for a page looks like. The run's figures were not kept, so this is the likely cause, not a demonstrated one; the test is unchanged. - New: `a_reply_whose_tail_waits_on_the_disk_is_read_once_the_tail_is_done`. It fails against the earlier rule on an idle host and passes with this one. - The oracle's own quiet rule now asks the same of each thread, read raw, so it is not fooled where the meter no longer is. ## Verification On the build Mac: - `cargo test -p sandbox-meter`: all pass idle, and in each of twelve full runs under a load past 160 (72 disk-wait round trips, all exact, none with a gap). - The process-tree tests of the crates that use this meter all pass: `jcp-gateway`, `proof-solvers`, `sandbox-proof`, `joule-code-remediate`. Neither original failure reproduced under CPU load alone, before or after the change; both were seen only on a host that was also short of memory. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 1 commit 2026-10-13 01:03:53 -04:00

sandbox-meter: a member waiting on the kernel in the middle of its work is not settled ... 80e254933c

The tree meter took a member for settled when two reads of its counts
agreed with no thread on a core or queued between them. A thread in a wait
that no signal ends (the kernel's own waits: a block being written, a page
being brought in, a lock) passes that rule: it is off its core and folded.
But it runs again when the kernel is done, with nothing arriving, so the
member is in the middle of its work. A stop taken there read the span
short by the rest of that work, whole, with no gap.

Measured (M4 Max, macOS 27.0.1): a server whose reply is followed by
twenty rounds of a 4 KiB write, an fsync and a short burn read short in 24
of 60 round trips, at 16 to 92% of the oracle's advance, each reading
whole. With the rule below it read exact in 60 of 60, and in every one of
72 more under a load past 160.

The rule: once `pti_numrunning` is zero the meter asks each thread's own
run state (`PROC_PIDLISTTHREADS`, then `PROC_PIDTHREADINFO`), and a member
with a thread in `TH_STATE_UNINTERRUPTIBLE` is not settled; it is walked
again, and is `TreeGap::Unsettled` if it never comes to rest inside the
bound. A thread asleep, or blocked reading a pipe, is not in that state (0
of 20 looks each; a thread in an fsync loop, 18 of 60), and none of an
idle host's threads was at 5 looks, so a server waiting for its next
request settles as before. Cost: the start of a span on a tree of 61
processes took 1.73 to 1.75 ms against 1.68 to 1.72 ms.

JCP SPEC 5.1 states the requirement, and names the second cause beside
"still running" in the gap it already has.

Tests, two of which failed on a loaded host and are why this was found:

- `a_worker_reaped_while_its_tree_is_walked_is_never_missed` asserted no
  reading is whole. One round trip of 21 in a CI run read the worker alive
  and off every core, counted its record, and was whole: the worker was
  read, not missed. The test now asserts its name: a whole reading holds
  the worker as a member.
- `a_server_that_reaps_its_worker_is_not_whole` failed its equality with
  the oracle once at a load past 100 on a host short of memory. That is
  what a member read while it waited for a page looks like; the run's
  figures were not kept, so this is the likely cause and not a shown one.
- New: `a_reply_whose_tail_waits_on_the_disk_is_read_once_the_tail_is_done`,
  which fails against the earlier rule on an idle host.
- The oracle's own quiet rule asks the same of each thread, read raw.

Verified on the build Mac: sandbox-meter's tests idle and twelve full runs
under load, and the process-tree tests of jcp-gateway, proof-solvers,
sandbox-proof and joule-code-remediate.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 1 commit 2026-10-13 02:25:42 -04:00

jcp-gateway, proof-solvers: the tree tests' oracles ask each thread's wait state too ... c15615c785

These two tests hold the tree meter to an oracle of their own, which took a
process for quiet with no thread on a core or queued. The meter now also
waits out a thread in a wait no signal ends, so an oracle without that rule
could read a member mid-work where the meter waits, and the two would
disagree for a reason that is the oracle's. Each oracle now asks every
thread's run state, read raw, as tree_energy's does.

Both pass on the build Mac: upstream_tree 7 of 7, z3_tree 5 of 5.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot merged commit f244bd33a3 into main 2026-10-13 03:13:53 -04:00

dcharlot referenced this pull request from a commit 2026-10-13 03:13:55 -04:00

Merge pull request 'sandbox-meter: a member waiting on the kernel in the middle of its work is not settled' (#230) from fix/tree-energy-reap-under-load into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!230

No description provided.