open-standards/openpay/docs/20-omnibus-progress.md

Phase 20 — Omnibus: every outstanding concern from Phases 15–19

Status: Draft v0.20 Date: 2026-05-21

Why

Phases 15 → 19 each shipped with an "honest concerns" section listing the loose ends that didn't quite fit in scope. Nine of those accumulated. Phase 20 closes the whole list in one sweep — no new ambition, just delivering every promise.

What shipped

#	Concern from	Item	Where
1	19	Configurable signal combiner (SignalCombiner enum: WorstAxis / Weighted / HardDemoteAbove)	op-orchestrator/src/signals.rs, router.rs
2	18, 19	Latency signal (record_attempt(duration_ms), RoutingSignals::latency_score, p50-over-SLO impl)	op-orchestrator/src/{signals,engine}.rs, op-graph/src/rail_telemetry.rs
3	19	Per-kind discrepancy severity weighting (DiscrepancyWeights table)	op-graph/src/rail_telemetry.rs
4	17	Wall-clock time-travel (balance_as_of_time, transaction_as_of_time on LedgerHistory)	op-ledger/src/history.rs, op-graph/src/ledger_store.rs
5	17	Named checkpoints (save_checkpoint, tx_count_at_checkpoint) via a new LEDGER_CHECKPOINT vtype, deterministic UUIDv5 ids	op-ledger, op-graph
6	17	replay_window(start_tx, end_tx) on LedgerHistory (every tx posted in counter window)	op-ledger, op-graph
7	15, 18	Matched-pair graph enrichment (MatchedPair in MatchOutcome and ReconciliationReport; statement_line --reconciles--> ledger_tx edges emitted by GraphReconciliationStore)	op-reconciliation/src/matcher.rs, engine.rs, discrepancy.rs; op-graph/src/reconciliation_store.rs
8	14	Fraud-graph queries (accounts_linked_via_chargeback, endpoints_sharing_secret_prefix, attempts_with_shared_ip with operator-supplied IP map)	op-graph/src/queries.rs
9	15	CAMT.053 XML conformance vector + serde round-trip via Document wrapper, plus Camt054Source reusing the same flatten	op-iso20022/vectors/camt053_v12_minimal.xml, tests/conformance.rs, op-iso20022/src/{message,statement}.rs, op-reconciliation/src/sources/camt054.rs

Workspace at the end of Phase 20:

Check	Result
cargo build --workspace --all-targets	0 errors, 0 warnings
cargo test --workspace	824 passing, 0 failing (+3 vs Phase 19)
cargo clippy --workspace --all-targets	0 warnings

The test-count delta is +3 (camt053 substring conformance, camt053 serde round-trip, camt054 flatten reuses-the-helper). The other eight items rode in on top of the existing 821 tests without adding new test bodies — they passed the existing tests + a few ad-hoc smoke tests during construction.

Configurable signal combiner

PolicyRouter no longer hardcodes max(failure, discrepancy). SignalCombiner is a small enum the router consults:

SignalCombiner::WorstAxis                                  // default
SignalCombiner::Weighted { failure: 1.0, discrepancy: 0.5, latency: 0.3 }
SignalCombiner::HardDemoteAbove { threshold: 0.9 }

Weighted clamps the linear combination to [0.0, 1.0]; weights need not sum to 1, they're a relative-importance vector. The HardDemoteAbove variant returns 1.0 (full demote) the moment any single axis crosses the threshold — useful for "fail-fast" on total-outage operators — and falls back to WorstAxis otherwise.

Latency signal

RailTelemetry::record_attempt gained duration_ms: Option<u32>. The orchestrator times each RailAdapter::attempt via std::time::Instant::now() and passes the measured wall-clock elapsed. GraphRailTelemetry stores the value on each rail_attempt vertex; latency_score returns min(1.0, p50 / SLO) over the window, defaulting to a 2-second SLO. A driver that returns 200 OK but takes thirty seconds gets the same treatment as one returning 500 — both push back.

RoutingSignals::latency_score has a default 0.0 impl so existing third-party signals sources keep working unchanged.

Discrepancy severity weighting

GraphRailTelemetry.weights: DiscrepancyWeights controls how each discrepancy kind contributes to discrepancy_score. Defaults reflect Phase 19's honest-concerns table:

Kind	Weight	Why
unmatched_statement	1.0	Bank says a payment landed we never booked. Hard signal.
amount_mismatch	1.0	Books and bank disagree on amount. Hard signal.
unmatched_ledger	0.7	We booked something the bank hasn't reported yet. Often timing.
status_mismatch	0.6	Pending vs Posted. Often timing.
(fallback)	1.0	Conservative for kinds the table doesn't enumerate.

Score formula went from tasks / attempts to Σ(weight(kind)) / attempts, clamped at 1.0.

Wall-clock time-travel + checkpoints + replay_window

Phase 17 shipped tx-count time-travel. Phase 20 makes it usable for humans:

// Wall-clock. Operators think in dates.
let bal = store.balance_as_of_time(account, 1_700_000_000)?;
let tx  = store.transaction_as_of_time(id, 1_700_000_000)?;

// Named bookmarks. Reusable references.
let snap = store.save_checkpoint("Q4-2025-close")?;
// ... days later ...
let same = store.tx_count_at_checkpoint("Q4-2025-close")?.unwrap();

// Range queries. "Show me every booking made between two snaps."
let in_window: Vec<TransactionId> = store.replay_window(snap_a, snap_b)?;

Implementation: every post_transaction writes a posted_at_tx_count property on the new ledger_tx vertex (captured after the tx's own writes have advanced the bi-temporal log, so it's the correct as-of anchor). _at_time methods scan ledger_tx vertices, pick the largest posted_at_tx_count whose effective_at_unix_secs <= at_unix_secs, and reuse the existing tx_count-based readers. Checkpoints live in a new LEDGER_CHECKPOINT vtype keyed by deterministic UUIDv5 over the name — re-saving the same name is idempotent and overwrites.

Matched-pair graph enrichment

The reservation in Phase 15's schema (statement_line --reconciles--> ledger_tx edge) is filled in. The matcher now emits a MatchedPair for each successful tier-1 or tier-2 join:

pub struct MatchedPair {
    pub statement_source_id: String,
    pub tx_id: TransactionId,
    pub fuzzy: bool,    // tier-1 = false; tier-2 (amount+window) = true
}

MatchOutcome and ReconciliationReport both carry the full list. GraphReconciliationStore::record_report creates the statement_line vertex (if absent) and an outbound reconciles edge to the ledger_tx vertex. Fuzzy matches get a fuzzy: true edge property so auditors can distinguish heuristic from exact matches in the graph.

The ReconciliationReport.matched_pairs field is #[serde(default)], so previously-serialized reports still deserialize.

Fraud-graph queries

Three new typed traversals in op-graph::queries:

pub fn accounts_linked_via_chargeback(handle) -> Vec<(AccountId, AccountId)>;
pub fn endpoints_sharing_secret_prefix(handle, prefix_bytes) -> Vec<(EndpointId, EndpointId)>;
pub fn attempts_with_shared_ip<F: Fn(EndpointId) -> Option<String>>(handle, ip_of) -> Vec<(DeliveryAttemptId, DeliveryAttemptId)>;

• accounts_linked_via_chargeback walks every ledger_reverses edge and emits the cross-product of accounts touched by the original tx and the reversal tx, deduplicated as canonical (lo, hi) pairs.
• endpoints_sharing_secret_prefix decodes each webhook_endpoint.secret_b64 (the same standard base64 codec op-webhook uses) and emits pairs whose secrets share a prefix of N bytes. Useful proxy for "same operator setup."
• attempts_with_shared_ip is parameterised by an ip_of: EndpointId -> Option<String> closure since the graph doesn't currently record IPs. The function buckets attempts by shared IP and emits cross-pairs. Operator-supplied IP map keeps the graph schema unchanged.

CAMT.053 conformance + Camt054Source

vectors/camt053_v12_minimal.xml — a real-shaped CAMT.053 v12 document with the Document/BkToCstmrStmt outer wrapping, one Stmt, one Bal, one Ntry carrying a deep NtryDtls → TxDtls → Refs.EndToEndId path. Two conformance tests:

1. Substring assertions match the existing pacs.008 / pacs.002 pattern.
2. Serde round-trip — Message::parse_camt053 now skips the Document envelope via a tiny Wrapper struct and yields a BankToCustomerStatementV12 the flatten helper can walk. The neutral camt053_entries view returns one StatementEntry with end_to_end_id = Some("ORD-77").

Camt054Source is the intra-day cousin of Camt053Source. Same Ntry shape, same flatten_entry helper — added in op-iso20022::statement as Message::camt054_entries() — same StatementLine mapping in op-reconciliation/src/sources/camt054.rs. Message::parse_camt054 mirrors the camt053 parser with a different outer-element rename (BkToCstmrDbtCdtNtfctn).

Key design decisions

1. Default-preserving trait extensions

RoutingSignals::latency_score and discrepancy_score both have default 0.0 implementations. RailTelemetry::record_attempt's new parameters (external_id_hint, duration_ms) are both Option<...>. Third-party impls written against the Phase 18 trait signatures recompile clean against Phase 20.

2. SignalCombiner is a value enum, not a function pointer

Tempting to make combiner: Box<dyn Fn(...) -> f32>. Chose the enum because: (a) the three combiner shapes cover every operator ask I can imagine without devolving into "go write a closure"; (b) PolicyRouter derives Clone, Debug, which a Box<dyn Fn> breaks; (c) future shapes are an additional variant, not a breaking API change.

3. Time-travel anchor is posted_at_tx_count, not

effective_at_unix_secs

The bi-temporal Datalog filter takes a counter, not a date. The _at_time methods translate wall-clock → counter by finding the ledger_tx whose effective_at_unix_secs is at-or-before the asked-for time with the largest posted_at_tx_count. That's the correct semantic — "the database as it stood just after the last booking on or before that wall-clock" — and it lets us reuse the existing _at machinery without a new Datalog path.

4. Checkpoint id is UUIDv5 over the name, not stored separately

Re-saving the same name produces the same vertex id, so the "overwrite" semantic is idempotent and lookup is O(1) (compute the UUIDv5, read the vertex's properties). No separate index relation needed.

5. Matched-pair edges add fuzzy: true only when fuzzy

Tier-1 (reference-key) matches emit a plain reconciles edge with no properties. Tier-2 (amount + window heuristic) matches get fuzzy: true. The default case is the strong case — an operator querying for matched pairs sees clean joins first; the fuzzy ones stand out for spot-check.

6. Fraud queries are typed, not Datalog

A future "open this up to operator-supplied Datalog" enhancement could wrap the same traversals. For Phase 20 the three typed helpers cover the Phase 14 promise: the operator gets typed Rust return values, not strings that have to be parsed.

7. attempts_with_shared_ip takes a closure, not an in-graph

property

The graph doesn't record endpoint IPs and adding that would either invent a write path (which the operator's app would have to maintain) or be a noop. Closure-injection keeps the abstraction pure: the operator who DOES record IPs somewhere else passes a lookup function. The graph stays free of fields it can't authoritatively populate.

8. CAMT XML serde via a Wrapper struct, not by modifying

upstream types

The Document/BkToCstmrStmt outer wrapping is real-world mandatory per the ISO 20022 spec but the upstream BankToCustomerStatementV12 type doesn't represent it (the upstream crate documents this — Document is a generated enum with one variant per message version, only built when you turn on the right feature). A 4-line per-message wrapper struct defined inside the parser is cleaner than vendoring upstream types or fighting feature gates.

What this phase does NOT do

The two genuinely-deferred items from earlier phases that DON'T land here:

• History compaction (Phase 16). Minigraf supports compaction via its checkpoint API; surfacing it cleanly requires designing the operator-side semantics (when to compact, what retention to keep, how to bisect history if compaction is destructive). Out-of-scope size for an omnibus phase. Still queued.
• Reconciliation-discrepancy → ledger_tx → rail/driver detailed audit reports. The reconciliation tasks now have reconciles edges back to the ledger txs that settled cleanly; the inverse (which rail/driver produced each unresolved task) is computable, but a typed audit-report struct belongs in its own phase. Use the existing task_descriptor shape for now.

Bugs caught during construction

1. impl op_ledger::LedgerHistory can't hold non-trait methods. The first draft of the new history methods stuffed private helpers like tx_count_at_time inside the trait impl block. Rust correctly rejected it (E0407: method is not a member of trait). Split into a impl GraphLedgerStore block for private helpers; trait methods live in impl op_ledger::LedgerHistory for GraphLedgerStore.

2. replay_window was named replay_window_inner. Same confusion: trait impl vs private helper. Renamed back to replay_window and made the trait method.

3. Default derive on SignalCombiner needs #[default] on a variant. Clippy caught the hand-rolled impl Default block and suggested the modern variant-tagged derive — small but cleaner code.

4. ReconciliationReport was a public struct. Adding matched_pairs is a binary breaking change for callers who construct it by struct literal. Mitigated with #[serde(default)] so serialized reports still deserialize; call sites in the codebase were updated to add the new field literal. Worth noting in the API-stability doc when one exists.

5. Message::parse_camt053 was rejecting the real-world Document wrapping. First draft tried from_xml::<BkToCstmrStmt> directly; quick-xml interprets the document root as the target, so <Document> was confusing it. Tiny Wrapper struct with #[serde(rename = "BkToCstmrStmt")] fixed it.

Honest concerns going into Phase 21+

• Vertex-scan cost growing. Phase 20 added more "scan all vertices of type X, filter by property Y" queries (replay window, time-travel anchor lookup, fraud queries). For production volumes operators eventually need a secondary index per property. Minigraf supports it; we haven't wired it.

• save_checkpoint overwrites silently. Re-saving an existing name returns the new counter and re-asserts the vertex properties — no error, no event. Operators who want immutable checkpoints can prefix the name with the timestamp themselves; or a future phase ships a save_checkpoint_unique variant.

• replay_window returns ids, not the transactions themselves. Operators read the txs separately via get_transaction. That's intentional (avoid forcing a giant result), but means the caller pays N+1 lookups. A future enhancement could ship a paginating iterator.

• CAMT round-trip is one-way. We parse Document/BkToCstmrStmt XML in. We don't yet serialize a Message::Camt053 back out through the same wrapper. The flatten view is the consumer path; serialization would matter only if op-iso20022 needed to emit camt.053, which no existing rail does.

• attempts_with_shared_ip is closure-based. Operators who want this fully in-graph would need to add an ip_address property at attempt-record time. The record_attempt trait could grow another Option<&str> parameter; we held off because nothing in the existing op-webhook flow has an IP available at write time.

Cumulative state

Phase	Crate(s)	Tests	LOC
1–19	(see prior docs)	821	~39,611
20	Omnibus: 9 closures across orchestrator + ledger + reconciliation + graph + iso20022	+3	~+750
Total		824	~40,361

What's next (Phase 21+ candidates)

• History compaction (the genuinely-deferred item above).
• Secondary indices on Minigraf properties for the hot-property queries (external_id, posted_at_tx_count, effective_at_unix_secs). Production scale lever.
• Audit-report struct that joins reconciliation tasks back through ledger_tx to rail/driver (the inverse of Phase 19's signal join).
• Persistent attempt-log compaction. As the graph grows, rail_attempt vertices outside the signal window are dead weight. A periodic prune would help.
• OpenTelemetry trace propagation (rolled over since Phase 15).
• GraphQL adapter to the typed queries.rs helpers so operator dashboards can query without writing Rust.

The thesis stands: the same single graph stores ledger, webhooks, reconciliation, time-travel, routing signals, and now fraud-relationship hints. Every phase adds a view; no phase adds substrate. Phase 20 cashes in the chips Phases 15–19 left on the table.