b77b9d70f6
All checks were successful
repository checks / lockfiles, advisories, tests in CI, workflow hygiene, spec and section references, public text (push) Successful in 11m52s
The loop was a flat queue of independent tasks routed by kind. A flat queue is a work pool, not automation: nothing could express "transform after fetch, and never publish because fetch failed". `Task::after` names the steps a task waits on, and only an item whose every dependency is `Done` is dispatched. An edge is one task id naming another in this run, resolved against the run's own journal — not an endpoint in someone else's system. That is the whole point of putting the graph here rather than in an integration fabric: the fabrics put their edges across trust boundaries they do not own, which is where their failures live, and an edge that does not cross a boundary cannot fail at one. A step whose dependency did not complete is settled `WorkState::Blocked`, naming what it waited on and why that did not happen. Blocked is deliberately not Denied: a denial is a decision about a grant and carries a sealed receipt, while a block is a missing precondition — no harness was offered the task, no grant was consulted, nothing was spent, so there is nothing to seal. The blocking pass runs to a fixed point, because settling one step is what determines why the next can never run; a single pass would have to call a long chain behind one failure a cycle, which it is not. A real cycle lands there too, so the loop still terminates on any graph a caller can submit. `Status::Retry` was taken at its word: the harness said transient, so the loop ran the task again. But transient describes why the attempt failed, not whether it left anything behind — a harness can fail after its effect has landed, and the second run then pays for and performs that effect twice. The crate already refuses that bet for `Interrupted`; it was taking it here. A retry is now honoured only for a task carrying `Task::idempotency_key`, and refused otherwise with that reason recorded. An attempt bound was never a substitute: it limits how many times an effect may be duplicated, it does not make duplicating it safe. SPEC §4.11 gains the rule. It already governed a decision recorded with no outcome; this is the neighbouring case its own reasoning implies — an outcome recorded as "run it again" — and it requires the key to be part of the durable record, so a restarted enforcement point can tell a replay-safe action from one that merely has attempts left. Both the graph edges and the replay identity are journalled with the task, so a resumed run rebuilds the graph and not a flat queue. The example is now a five-step pipeline, submitted in reverse so the ordering cannot be an artefact of the queue: the ship is denied for want of a capability and the announcement behind it is never started. jouledesk gains the matching `TagOutcome::Blocked`, which seals no efficiency receipt — there were no joules to account for. Co-Authored-By: Claude Opus 5 <[email protected]>
179 lines
12 KiB
Markdown
179 lines
12 KiB
Markdown
# The background-agent orchestrator pattern → `jcp-orchestrate`
|
||
|
||
`jcp-orchestrate` does not invent an orchestration shape. It **ingests** the one
|
||
the agent-work ecosystem converged on — a background-agent orchestrator that runs
|
||
coding agents in the cloud — and supplies the three things every deployed
|
||
instance of that pattern lacks: a **capability gate**, a **joule budget**, and a
|
||
**sealed audit**.
|
||
|
||
This document maps the pattern — as built by the production background-agent
|
||
orchestrators — onto (1) this crate's reconcile loop, (2) the Sandbox S0–S7
|
||
isolation ladder its container model sits low on, (3) the Joule Table seams its UX
|
||
surfaces never carried, and (4) the JCP grant its credential model stops short of
|
||
— then scopes the frontier the whole category leaves empty.
|
||
|
||
## 0. Why map a commodity pattern
|
||
|
||
The background-agent orchestrator is a well-worn shape: a tracker of work items;
|
||
each item run by a coding agent in an isolated container on its own repo copy,
|
||
auto-branched, checkpointed to a forge as machine-authored commits and pull
|
||
requests; a CLI for local handoff that doubles as an MCP server; bring-your-own
|
||
subscription or API keys; recurring and event-triggered automations; chat and
|
||
forge @-mention bindings. Several products shipped exactly this. The shape is not
|
||
in dispute and not a moat — it is commodity.
|
||
|
||
What none of them produced is the durable artifact: a **signed, energy-accounted,
|
||
verifiable receipt** of what ran, under whose authority, for how many joules.
|
||
Everyone can build the orchestrator; the accountability is the open lane. This
|
||
crate keeps the loop and makes each dispatch a JCP decision made **outside** the
|
||
agent.
|
||
|
||
## 1. The reconcile loop → `Orchestrator` (ingest, not compete)
|
||
|
||
The pattern's control plane is a tracker → route → dispatch → retry loop over a
|
||
*heterogeneous* fleet, where a worker is one coding-agent backend on one item and
|
||
a different backend on the next. `Orchestrator` keeps that loop and gates every
|
||
dispatch.
|
||
|
||
| Background-agent orchestrator concept | `jcp-orchestrate` symbol | What TX adds |
|
||
|---|---|---|
|
||
| Work tracker, one branch per task | `Task { id, kind, payload, blocked_by, idempotency_key }` + the sealed decision ledger (`receipts()`) | The run is an **auditable ledger of sealed decisions**, not just a branch list |
|
||
| A coding-agent backend (any CLI/framework) | `Harness` — any backend that quotes a joule cost and reports an outcome | The orchestrator is **blind to what runs inside**; backends are `register`ed, not hardcoded |
|
||
| Assign a task to a backend | `dispatch` **decided against a grant, outside the backend** (`route` → gate) | Who-runs-what is a runtime authorization, never the agent's own choice |
|
||
| Retry on failure | bounded retry **+ failover through a sealed denial** to a backend still under budget | An exhausted backend is denied *on the record*, then fails over; a flaky one retries to `max_attempts` |
|
||
| Bring-your-own subscription / API key | `Capability` (task kinds) **+ `budget_uj`** per backend | Authority, not just authentication — see §4 |
|
||
| Unbounded agent loop (the documented cost failure) | joule budget **debited + enforced before the side effect** | A hard, pre-side-effect energy ceiling per grant |
|
||
| Machine-authored commit / PR as the record | JCR-1 receipt sealed per decision, allow **and** deny | The **non-repudiable audit** a git history cannot be |
|
||
|
||
Net: the pattern is *what the loop looks like in production*; `Orchestrator` is
|
||
*the same loop with the gate, the budget, and the sealed audit the meta-harness
|
||
leaves to the platform*. The `background_agents` example drives it end to end.
|
||
|
||
**What is and isn't sealed.** Every decision made *against a grant* is sealed —
|
||
the allow, and the deny when a backend is out of budget or out of scope. Budget
|
||
enforcement in particular must leave an artifact: a joule ceiling that fires
|
||
silently is an enforcement nobody can audit, so a backend that cannot pay is
|
||
denied through the runtime and that denial is on the record *before* the loop
|
||
fails over to the next. The one refusal that seals nothing is a task whose kind
|
||
**no** registered backend holds: there is no grant to decide against, and JCP
|
||
seals decisions about a grant, not the absence of one. Such an item is recorded
|
||
denied in the `RunReport` and never reaches a backend.
|
||
|
||
## 2. Container isolation → Sandbox S0–S7 (attested execution)
|
||
|
||
The pattern isolates each task in a container with its own repo copy. On the
|
||
Sandbox ladder that is roughly **S2–S3**: namespace/cgroup isolation, no attested
|
||
boot, no measured execution, no energy-counter provenance. The container keeps
|
||
tasks from colliding; it proves nothing to a third party about what ran.
|
||
|
||
| Pattern isolation | Sandbox rung | The step up |
|
||
|---|---|---|
|
||
| Container, own repo copy | ~S2/S3 (namespace isolation) | Integrity asserted by the host, attested to no one |
|
||
| — | **S5** — microVM (proven on real bare-metal KVM + RAPL) | A hardware-isolation domain with a measured boot; a receipt can name the image that ran |
|
||
| — | **S6** — TEE attestation (`sandbox-tee::energy`) | An `EnergyStatement` bound into a TEE report's `report_data`; the honest `IntegrityAttested` vs `PhysicallyRooted` trust ladder |
|
||
|
||
The move is not a better container. It is running the *same* task inside an S5
|
||
microVM under an attested boot, and binding the energy reading's provenance into
|
||
the receipt. The pattern's isolation stops at "tasks don't collide"; the ladder
|
||
continues to "a third party can verify what ran and what it cost."
|
||
|
||
## 3. The UX surfaces → Joule Table seams (gate / meter / receipt)
|
||
|
||
The surfaces the pattern converged on are the right surfaces — they are simply
|
||
unmetered, ungated, and unsealed. Each rebinds through an existing binding, and
|
||
each gains the three seams the bare orchestrator never had.
|
||
|
||
| Pattern surface | Joule Table binding | The three seams it gains |
|
||
|---|---|---|
|
||
| Local-handoff CLI + MCP task server | **Joule Hook** (editor/host binding) | gate = grant on task creation; meter = joules per task; receipt = MCP receipt server |
|
||
| Chat / forge @-mention kicks off a task | **Joule Desk** (teammate binding) | signed-grant authority for the mention; a per-receipt-verifiable ledger of what the teammate did |
|
||
| Recurring + event-triggered automations | **Durable agent execution** (Sandbox + JCP) | durable resumable runs, env-manifest identity, standing grants with a joule ceiling, mid-run steering |
|
||
| Browser streaming / notifications | mid-run steering over an attested run | monitoring becomes attested control |
|
||
|
||
The pattern proved the surfaces convert. It never made any of those actions
|
||
accountable. The seams are exactly that missing accountability.
|
||
|
||
## 4. Credentials → JCP grant (authentication → authority)
|
||
|
||
Bring-your-own subscription answers **who is paying** — an identity with a quota
|
||
at a vendor. A JCP grant answers **what this principal may do, for how many
|
||
joules, decided where the model can't reach**: a capability and an energy budget
|
||
are the same signed, bounded, debited object. A vendor key lets an agent run
|
||
until the bill stops it; a grant denies the action itself — an out-of-scope task
|
||
kind, or one over the joule ceiling — *before* the side effect, and seals both the
|
||
allow and the deny. Auth says "the card works"; the grant says "this action, this
|
||
budget, this receipt."
|
||
|
||
## 4a. The integration fabric → edges inside one stack
|
||
|
||
The adjacent category is the workflow-automation tool: a canvas of nodes wired
|
||
into a graph, where each node reaches an endpoint in a different company's
|
||
system. Those products are worth taking seriously on shape and not on structure.
|
||
Their value *is* the bridging, so their failure surface is the bridges — a
|
||
renamed field, a rotated credential, a partial write, a rate limit mid-run, an
|
||
approval one system believes the other checked. The dominant open tool in the
|
||
category drew 237 published advisories in eighteen months, 22 of them critical,
|
||
including five approval-forgery or bypass advisories in a single month. That is
|
||
not a quality problem to out-engineer; it is what a graph whose edges cross trust
|
||
boundaries nobody owns costs to run.
|
||
|
||
`jcp-orchestrate` takes the graph and not the bridges. `Task::after` names the
|
||
steps a task waits on, and an edge is **one task id naming another in this run**,
|
||
resolved against the run's own journal. A step that waits on a step that did not
|
||
complete is settled `Blocked`, naming what it waited on — distinct from `Denied`,
|
||
which asserts a decision about a grant was made and sealed. Nothing crosses a
|
||
boundary this stack does not own, so the whole class of boundary error has
|
||
nowhere to occur, and the approval that gates a dispatch is a signed grant
|
||
decided outside the harness rather than a message one service hopes another
|
||
validated.
|
||
|
||
| Integration-fabric concept | `jcp-orchestrate` symbol | What changes |
|
||
|---|---|---|
|
||
| Node graph with branch/merge | `Task::after` → dependency edges, `WorkState::Blocked` | Edges name tasks in this run, not third-party endpoints |
|
||
| Retry on a failed node | `Status::Retry` gated by `Task::idempotency_key` | A retry needs a name for the work (SPEC §4.11); without one it is refused, not risked |
|
||
| ~400 app connectors + hosted OAuth broker | *deliberately absent* | Connectors are product surface, not a standards problem; the stack's own capabilities are first-class harnesses |
|
||
| Approval step in the canvas | JCP grant decided outside the harness, sealed allow **and** deny | An approval that cannot be forged between two services, because there are not two services |
|
||
|
||
The connector count is the category's moat and is not contested here. What is
|
||
contested is the premise that automation must be assembled out of other people's
|
||
interfaces at all.
|
||
|
||
## 5. The frontier the pattern leaves open
|
||
|
||
The background-agent orchestrators shipped every convenience and none of the
|
||
trust layer. The empty lane is the one no instance of the pattern occupies: a
|
||
**per-run, per-task, energy-accounted, attested, third-party-verifiable receipt**
|
||
of agentic work — `efficiency/1` rolled up from JCR-1 decisions, with the energy
|
||
counter's provenance attested through `sandbox-tee`. No competitor issues it;
|
||
regulation mandates nothing per-task; the category publishes rates, not
|
||
outcome-anchored receipts.
|
||
|
||
## 6. A conformant flow (the profile in one paragraph)
|
||
|
||
Register each coding-agent backend as a `Harness` with a **capability** (the task
|
||
kinds it may handle) and a **joule budget**. Track work items as the pattern does;
|
||
run the reconcile loop, but make every dispatch a JCP decision against that grant,
|
||
**sealed as JCR-1 before the task runs**. Run the task inside a **Sandbox S5
|
||
microVM** under an attested boot, with the energy reading's provenance bound via
|
||
`sandbox-tee::energy`. Roll the per-task receipts up with `aggregate()` into an
|
||
`efficiency/1` figure (`uj_per_accepted_token`, waste share, outcome anchor).
|
||
Rebind the surfaces — local-handoff/MCP through **Joule Hook**, chat/forge through
|
||
**Joule Desk**, automations through **Durable agent execution** — each carrying
|
||
gate / meter / receipt. The result is the pattern's exact UX, with the one thing
|
||
it structurally could not produce: a signed, energy-accounted, verifiable record
|
||
of every action.
|
||
|
||
## Status
|
||
|
||
- Reconcile loop + capability/budget gate + sealed ledger — shipped (`src/lib.rs`).
|
||
- Step dependencies (`Task::after`, `WorkState::Blocked`) and the idempotency
|
||
precondition on retry (SPEC §4.11) — shipped (`src/lib.rs`), with the graph
|
||
edges and the replay identity carried in the journal so a resumed run rebuilds
|
||
the graph and not a flat queue.
|
||
- End-to-end demonstration — shipped (`examples/background_agents.rs`), a
|
||
five-step pipeline where the ship is denied for want of a capability and the
|
||
announcement behind it is never started.
|
||
- Sandbox rung integration (run the harness inside an S5 microVM) and the
|
||
`efficiency/1` roll-up — the composition this document scopes; the primitives
|
||
(`sandbox-tee::energy`, `jcp-efficiency::aggregate`) are already shipped in their
|
||
own crates.
|