jouleclaw-graph: a node in flight survives the process, and an interrupted one is held rather than silently run again #333

Merged

dcharlot merged 1 commit from fix/jouleclaw-graph-dispatch-before-run into main 2026-10-19 06:08:23 -04:00

Conversation 0 Commits 1 Files changed 3 +400 -2

dcharlot commented 2026-10-19 02:55:07 -04:00

Owner

Copy link

The defect

A JouleClaw run journal (jouleclaw-graph) recorded a node only when it completed (NodeRecorded). Nothing recorded that a node had begun. A process that died in the middle of a node therefore left no trace of it, and a restarted run found the node ready again:

• Silent repeat: the next dispatch repeated whatever the node had done outside the run (a message sent, a payment made, a record written), with no sign it might already have happened.
• Lost energy: the interrupted attempt's energy was never recorded.

JCP §4.11 requires the decision to be recorded before the action runs and forbids silently retrying one whose outcome is unknown. jcp-orchestrate, mesh and the jcp/1 endpoint already do this; the run journal did not.

What prompted it: a comparison with durable agent harnesses. The common shape commits a tool call's intent before running it, reruns calls declared safe, and tells the model an unsafe one "may have partly run". Published crash tests of that default show its weakness: a model retried an interrupted write it could not see and left a duplicate. Here, an interrupted never node is not left to a model. It is held until someone settles it.

The fix

• Replay rule: Replay { Safe, Never }, with Never the default.
• New RunEvents:
  • NodeDispatched { attempt, replay }, written before work begins;
  • NodeInterrupted { attempt, observed_uj };
  • RerunApproved { approved_by }.
• Hashes unchanged: the rule travels with the dispatch, not on Node, so every graph's content address and every persisted journal are unchanged.
• Run::dispatch records the dispatch and returns the attempt number. A node in flight is not ready, so it is not dispatched twice.
• Run::recover marks every node still in flight after a restart as interrupted, keeping the energy observed for the attempt as a lower bound.
  • An interrupted Safe node is ready again, as a new attempt.
  • A Never node is held (held_nodes) until it is recorded, approved for a rerun by a named party (approve_rerun), or the run fails.
• Fail-safe: a consumer that omits recover leaves the node in flight and out of ready_nodes, so it stalls rather than repeating work. record without a prior dispatch works as before, so existing consumers are unaffected.
• jouleclaw-stuck counts dispatch, interruption and approval timestamps as activity, so a run whose last event is a long node's dispatch is not reported abandoned.
• JouleClaw SPEC §17.3 states it.

Checked

On the build host:

• Tests: jouleclaw-graph passes 47, six of them new. jouleclaw-stuck passes 17, one of them new.
• Consumers: jouleclaw-gate, graph-mcp, mission, artifact, reflection and promote pass. The JouleClaw workspace checks, and clippy is clean.
• Mutations: each fails tests:
  • ready_nodes ignoring in-flight and held nodes
  • dispatch ignoring a held node
  • every interrupted node rerunnable
  • interrupted energy dropped
  • stuck ignoring dispatch
• Repository checks: tests-in-CI, spec references, vanity metrics and orphan sources pass locally.

Not in this change

jouleclaw-mission dispatches features rather than graph nodes and has no resume of its own. Wiring its workers through dispatch / recover is the next step.

🤖 Generated with Claude Code

## The defect A JouleClaw run journal (`jouleclaw-graph`) recorded a node only when it **completed** (`NodeRecorded`). Nothing recorded that a node had begun. A process that died in the middle of a node therefore left no trace of it, and a restarted run found the node ready again: - **Silent repeat:** the next dispatch repeated whatever the node had done outside the run (a message sent, a payment made, a record written), with no sign it might already have happened. - **Lost energy:** the interrupted attempt's energy was never recorded. JCP §4.11 requires the decision to be recorded before the action runs and forbids silently retrying one whose outcome is unknown. jcp-orchestrate, mesh and the `jcp/1` endpoint already do this; the run journal did not. **What prompted it:** a comparison with durable agent harnesses. The common shape commits a tool call's intent before running it, reruns calls declared `safe`, and tells the model an `unsafe` one "may have partly run". Published crash tests of that default show its weakness: a model retried an interrupted write it could not see and left a duplicate. Here, an interrupted `never` node is not left to a model. It is held until someone settles it. ## The fix - **Replay rule:** `Replay { Safe, Never }`, with `Never` the default. - **New `RunEvent`s:** - `NodeDispatched { attempt, replay }`, written **before** work begins; - `NodeInterrupted { attempt, observed_uj }`; - `RerunApproved { approved_by }`. - **Hashes unchanged:** the rule travels with the dispatch, not on `Node`, so every graph's content address and every persisted journal are unchanged. - **`Run::dispatch`** records the dispatch and returns the attempt number. A node in flight is not ready, so it is not dispatched twice. - **`Run::recover`** marks every node still in flight after a restart as interrupted, keeping the energy observed for the attempt as a lower bound. - An interrupted `Safe` node is ready again, as a new attempt. - A `Never` node is **held** (`held_nodes`) until it is recorded, approved for a rerun by a named party (`approve_rerun`), or the run fails. - **Fail-safe:** a consumer that omits `recover` leaves the node in flight and out of `ready_nodes`, so it stalls rather than repeating work. `record` without a prior `dispatch` works as before, so existing consumers are unaffected. - **jouleclaw-stuck** counts dispatch, interruption and approval timestamps as activity, so a run whose last event is a long node's dispatch is not reported abandoned. - **JouleClaw SPEC §17.3** states it. ## Checked On the build host: - **Tests:** jouleclaw-graph passes 47, six of them new. jouleclaw-stuck passes 17, one of them new. - **Consumers:** jouleclaw-gate, graph-mcp, mission, artifact, reflection and promote pass. The JouleClaw workspace checks, and clippy is clean. - **Mutations:** each fails tests: - `ready_nodes` ignoring in-flight and held nodes - `dispatch` ignoring a held node - every interrupted node rerunnable - interrupted energy dropped - stuck ignoring dispatch - **Repository checks:** tests-in-CI, spec references, vanity metrics and orphan sources pass locally. ## Not in this change `jouleclaw-mission` dispatches *features* rather than graph nodes and has no resume of its own. Wiring its workers through `dispatch` / `recover` is the next step. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot force-pushed fix/jouleclaw-graph-dispatch-before-run from 4fa2a0d6f2 to 35b7bd3972 2026-10-19 02:56:08 -04:00 Compare

dcharlot merged commit a7f2352d27 into main 2026-10-19 06:08:23 -04:00

dcharlot referenced this pull request from a commit 2026-10-19 06:08:24 -04:00

Merge pull request 'jouleclaw-graph: a node in flight survives the process, and an interrupted one is held rather than silently run again' (#333) from fix/jouleclaw-graph-dispatch-before-run into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!333

No description provided.