open-standards/openpay/docs/19-discrepancy-signal-progress.md

Phase 19 — Reconciliation-density routing signal

Status: Draft v0.19 Date: 2026-05-21

Why

Phase 18's failure_score only catches drivers that fail at the HTTP edge: timeouts, 5xx, transport errors. A worse class of failure is silent: the adapter happily returns 200, the operator books the ledger transaction, and then — days later — the bank statement disagrees. PSP fee netting wrong, settlement amount drift, double-credit on a refund. These are the discrepancies Phase 15 already classifies into typed reconciliation_task vertices.

Phase 19 makes the router use those tasks. The signal:

recent rail_attempts(rail, driver)
  ↳ ledger_tx whose external_id matches an attempt's idempotency key
      ↳ inbound task_about edges = how many discrepancies touch this work

A driver whose transactions accumulate discrepancies gets pushed back in the fallback chain — even though every one of its HTTP attempts returned OK.

What shipped

A new method on RoutingSignals, an extended RailTelemetry signature, and the join logic in GraphRailTelemetry. The router combines both scores; the orchestrator threads the idempotency key through as a correlation token.

File	LOC	Notes
op-orchestrator/src/signals.rs	+25	RoutingSignals::discrepancy_score (default 0.0); RailTelemetry::record_attempt gains external_id_hint: Option<&str>
op-orchestrator/src/router.rs	+20	combined_score = max(failure, discrepancy); PolicyRouter sort uses it
op-orchestrator/src/engine.rs	+3	pass intent.idempotency_key.as_str() to record_attempt
op-graph/src/rail_telemetry.rs	+90	store external_id_hint property; new discrepancy_score + discrepancy_score_at impls; RailAttemptRecord.external_id_hint field
op-graph/tests/graph_routing.rs	+2	update Phase 18 test to new record_attempt signature
op-graph/tests/discrepancy_routing.rs	213	new — 3 end-to-end tests
Phase 19 totals	~355	net new

Workspace at the end of Phase 19:

Check	Result
cargo build --workspace --all-targets	0 errors, 0 warnings
cargo test --workspace	821 passing, 0 failing (+3 vs Phase 18)
cargo clippy --workspace --all-targets	0 warnings

The graph join

            ┌────────────────────────────────────────────────────┐
            │           GraphRailTelemetry::discrepancy_score    │
            └────────────────────────────────────────────────────┘
                                    │
                                    ▼
        rail_attempt vertices (window: now - window_secs)
          • filter by (rail, driver)
          • collect external_id_hint values
                                    │
                                    ▼
        ledger_tx vertices whose external_id ∈ collected hints
          • the "what work did this driver actually do?"
                                    │
                                    ▼
        inbound TASK_ABOUT edges on those ledger_tx vertices
          • each one is a reconciliation_task pointing at this tx
          • the discrepancy "count" for this driver
                                    │
                                    ▼
        score = min(1.0, tasks / attempts)

external_id_hint is the join key. In production it's the intent's idempotency key — most deployments propagate that verbatim to the ledger transaction's external_id. We don't enforce that propagation; we just make it work when operators follow that convention (which the existing op-orchestrator / op-ledger examples do).

The router math

combined_score = max(failure_score, discrepancy_score). Both sit in [0.0, 1.0]; the worst-news-in-either-dimension is the ranking key. A clean-HTTP-but-noisy-reconciliation driver still gets pushed back; a failing-HTTP-but-no-recon-history driver still gets pushed back. The two dimensions aren't additive — a broken driver isn't doubly broken — but either alone is enough to demote it.

Stable sort on tied scores still preserves operator-configured preference, same as Phase 18.

Key design decisions

1. Default discrepancy_score = 0.0

The trait method has a default body returning 0.0, so NoOpRoutingSignals and any third-party impl that doesn't track reconciliation state needs zero work. Implementations that DO have access can override. Same pattern as the set_endpoint_metadata default-empty trait methods elsewhere in the codebase.

2. external_id_hint is Option<&str>, not required

A few attempt sites legitimately have no ledger correlation — a dry-run probe, a health check, a manual replay that doesn't post. The trait accepts None and GraphRailTelemetry simply omits the property. Those attempts contribute to failure_score but never to discrepancy_score, which is the correct semantic.

3. Cap the score at 1.0

A particularly broken driver might have 5 reconciliation tasks against 2 attempts (e.g. several days of statement-line mismatches against the same buggy charge). The raw tasks / attempts could exceed 1.0; we clamp. The router only needs an ordering anyway, not a probability — but the docs commit to [0.0, 1.0] and we keep that promise. Anything north of 1.0 just means "as bad as it gets, push back."

4. Walk-the-vertices, not a stored index

The join scans rail_attempt and ledger_tx vertices on every discrepancy_score call. No secondary index. This is the same operational profile as failure_score, and at reference-impl scale (hundreds or low thousands of recent attempts in the window) it's plenty fast. A production deployment with millions of attempts would either (a) shrink the window, (b) shrink history via compaction, or (c) ship a custom RoutingSignals that consults an external index. The trait shapes for all three.

5. Idempotency-key propagation is the only contract

The new join works because operators conventionally set their ledger transaction's external_id to the same idempotency key the intent carried. We don't enforce that — the docs do. If an operator's ledger writes use a different correlation token, they ship their own RailTelemetry impl that records that token instead. The orchestrator just forwards what's on the intent.

6. The discrepancy signal lives alongside, not instead of,

the failure signal

Phase 18's failure_score is unchanged. Phase 19 adds a second axis. The router takes the max. This composition keeps each signal independently meaningful (and independently debuggable in operator dashboards) — operators can ask "is this driver bad because it's failing HTTP or because its books are wrong?" and the two scores answer separately.

What the end-to-end test demonstrates

crates/op-graph/tests/discrepancy_routing.rs:

1. Static config: [dirty_psp, clean_psp] in card_drivers.
2. Both adapters always return Success — Phase 18's signal is silent.
3. First intent ("intent-1"): chain is [dirty, clean], dirty tried first, succeeds. One attempt recorded against dirty with external_id_hint = "intent-1".
4. Operator posts a ledger transaction with external_id = "intent-1". (Test simulates the production pattern.)
5. Reconciliation records a UnmatchedLedger task pointing at that ledger_tx.
6. discrepancy_score(card, dirty) = 1.0 — one task per one attempt.
7. Second intent: combined score reorders chain to [clean, dirty]. Clean tried first, succeeds. One attempt, dirty never touched.

Plus two negative tests: empty reconciliation state yields score 0.0; a driver with no attempts in window yields 0.0.

What this phase does NOT do

• No tagging ledger_tx with the rail/driver directly. That would shorten the join — reconciliation_task → ledger_tx → rail/driver instead of three hops — but it would mean the ledger writer (which today is operator app code) has to know the rail/driver. The orchestrator doesn't post ledger txs; the operator's app does, often based on the orchestrator's outcome. Phase 19 deliberately stays out of that boundary; the join goes through external_id because that's the operator- controlled coupling that already exists.

• No per-task severity weighting. An UnmatchedLedger and an AmountMismatch count the same. Operators who care can ship a custom RoutingSignals that walks reconciliation_task.kind and weights.

• No discrepancy-task aging. A reconciliation task from three weeks ago counts the same as one from today, as long as the rail_attempt that produced it is in the current window. Window-scoping is on rail_attempt, not the task. If operators care about task-age decay, that's a custom impl.

• No automatic discrepancy resolution. Phase 19 reads tasks to route around them; resolving them is still operator work (mark them resolved, retract the vertex, etc.).

Bugs caught during construction

1. record_attempt signature drift broke Phase 18's persistence test (graph_routing.rs). Two callsites needed None for the new external_id_hint parameter. cargo pointed straight at them; mechanical fix.

2. etypes not imported in rail_telemetry.rs after the new in_edges(.., etypes::TASK_ABOUT) call. Added to the import line. The same use crate::graph::{etypes, vtypes, GraphHandle} pattern that every other store uses.

3. u32::try_from(...).unwrap_or(...) for the task count — could overflow u16 and the score would be inf. The min(1.0) cap catches it but the cast was still wrong; replaced with the same safe-cast idiom Phase 18 used.

Honest concerns going into Phase 20+

• Task age is unbounded by the window. As noted above, a task from a year ago still bumps the score if the rail_attempt that produced it is in the window — unlikely, but possible if the attempt log is huge and the window is huge.

• The join is O(rail_attempts × ledger_tx + tasks per candidate tx). For reference-impl volumes this is invisible; for production it's the same scale concern Phase 18 already documented.

• external_id is a property, not an indexed attribute in the current Minigraf usage. We scan all ledger_tx vertices and string-compare. A secondary EAV index keyed by external_id value (Minigraf supports it) would speed this to O(hints) lookups. Not pulled in yet; trivial to add when performance forces the issue.

• combined_score = max is a choice, not a derivation. Operators might want weighted (0.7 failure + 0.3 discrepancy) or threshold (failure if > 0.5 else discrepancy). The current router is hardcoded to max; future enhancement could expose a combined: fn(f32, f32) -> f32 knob.

Cumulative state

Phase	Crate(s)	Tests	LOC
1–18	(see prior docs)	818	~39,256
19	discrepancy_score on RoutingSignals + GraphRailTelemetry join	+3	+355
Total		821	~39,611

What's next (Phase 20+ candidates)

• Wall-clock time-travel + named checkpoints (still queued from Phase 17).
• Latency signal. Add duration_ms to record_attempt, surface it as a third RoutingSignals axis.
• CAMT.053 XML conformance + Camt054Source (still queued from Phase 15).
• Weighted / configurable signal combiner. combined_score becomes a function the operator can swap (max, weighted, threshold-based).
• History compaction (rolled over from Phase 16). Now even more relevant: the discrepancy join walks the rail_attempt log alongside the ledger / reconciliation graphs.
• Reconciliation-task severity weighting. Per-kind weights in the discrepancy_score join.

The thesis stands: the same single-file graph stores attempts, ledger txs, reconciliation tasks, and time-travel history; each phase pulls a new view out of the same substrate. Phase 19's view is the one that catches the silent failures — the ones where the rail says yes and the books say no — and routes around them automatically.