70ecd6caa0
Transaction Science open standards monorepo. Each subdirectory is a self-contained reference protocol with its own README and licence. - openpay/ Apache-2.0 payment-acceptance stack (Rust, 26 crates) - smart-byte/ CC-BY-4.0 content-addressed value carrier (spec) - eoc/ CC-BY-4.0 energy-optimised compute (spec) The wire format and the right to fork are public. The optional hosted services are the offer; the protocols are not.
441 lines
20 KiB
Markdown
441 lines
20 KiB
Markdown
# Phase 15 — `op-reconciliation` ledger vs. statement diffing
|
|
|
|
**Status**: Draft v0.15
|
|
**Date**: 2026-05-19
|
|
|
|
## Why reconciliation is non-negotiable
|
|
|
|
Every payment stack needs reconciliation because the ledger and the
|
|
bank/PSP record of the same money inevitably drift:
|
|
|
|
- A rail confirms settlement but the webhook never arrives, so the
|
|
ledger never posts.
|
|
- A chargeback debits the merchant account at the bank but no
|
|
reversal is booked.
|
|
- A PSP fee is netted out of a payout and the gross/fee split was
|
|
recorded wrong.
|
|
|
|
Without reconciliation these are invisible until an auditor or an
|
|
angry customer finds them. With it, each one becomes a typed
|
|
`Discrepancy` an operator can route to a ticketing queue.
|
|
|
|
Per the foundation doc, "bank-statement reconciliation is a set
|
|
difference, audits are a SQL query." Phase 15 makes that literally
|
|
true for OpenPay deployments.
|
|
|
|
## What shipped
|
|
|
|
A new top-level crate `crates/op-reconciliation`, neutral statement
|
|
plumbing in `op-iso20022`, and a graph-backed task store in
|
|
`op-graph`. **Total: 105 tests (97 unit + 5 + 1 integration + minor
|
|
spot tests), ~1,916 LOC** of new code. Cumulative across all 15
|
|
phases (revised — see "Baseline detour" below): **~963 tests,
|
|
~38,556 LOC**.
|
|
|
|
| File | Tests | LOC |
|
|
|---|---|---|
|
|
| `op-reconciliation/src/lib.rs` | — | 86 |
|
|
| `op-reconciliation/src/error.rs` | — | 56 |
|
|
| `op-reconciliation/src/statement.rs` | 2 | 139 |
|
|
| `op-reconciliation/src/source.rs` | — | 25 |
|
|
| `op-reconciliation/src/discrepancy.rs` | — | 175 |
|
|
| `op-reconciliation/src/matcher.rs` | 10 | 382 |
|
|
| `op-reconciliation/src/engine.rs` | 2 | 144 |
|
|
| `op-reconciliation/src/store.rs` | — | 44 |
|
|
| `op-reconciliation/src/sources/mod.rs` | 3 | 132 |
|
|
| `op-reconciliation/src/sources/camt053.rs` | — | 90 |
|
|
| `op-reconciliation/src/sources/webhook.rs` | — | 94 |
|
|
| `op-reconciliation/tests/integration.rs` | 5 | 168 |
|
|
| `op-iso20022/src/statement.rs` (new) | 2 | 185 |
|
|
| `op-graph/src/reconciliation_store.rs` (new) | — | 196 |
|
|
| **Phase 15 totals** | **24** | **~1,916** |
|
|
|
|
Plus 1 new cross-domain test in `op-graph/tests/integration.rs`
|
|
(reconciliation_tasks_wire_into_the_shared_ledger_graph), so the
|
|
phase nets **25 new tests**.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
crates/op-reconciliation/
|
|
├── Cargo.toml — deps: op-core, op-ledger, op-webhook,
|
|
│ op-iso20022, time (date parsing)
|
|
├── src/
|
|
│ ├── lib.rs — module roots + doc
|
|
│ ├── error.rs — sealed Error (6 variants) + Iso20022/Json bridges
|
|
│ ├── statement.rs — StatementLine, LineDirection (kept distinct from
|
|
│ │ op_ledger::Direction)
|
|
│ ├── source.rs — ReconciliationSource trait (sync iterator)
|
|
│ ├── discrepancy.rs — Discrepancy enum (4 variants),
|
|
│ │ ReconciliationReport, TaskDescriptor
|
|
│ ├── matcher.rs — two-tier match: external_id join then
|
|
│ │ amount+window heuristic
|
|
│ ├── engine.rs — Reconciler::reconcile(source, ledger_txs)
|
|
│ ├── store.rs — ReconciliationStore trait + ReconciliationTask
|
|
│ └── sources/
|
|
│ ├── mod.rs — shared codecs: currency, money, date
|
|
│ ├── camt053.rs — Camt053Source
|
|
│ └── webhook.rs — WebhookEventSource + reference JSON schema
|
|
└── tests/integration.rs — webhook end-to-end scenarios
|
|
|
|
crates/op-iso20022/ (additions)
|
|
├── src/message.rs — Message::Camt053 variant + parse_camt053 +
|
|
│ as_camt053
|
|
├── src/statement.rs — NEW: StatementEntry neutral view +
|
|
│ Message::camt053_entries flattening (walks
|
|
│ AccountStatement13 → ReportEntry14 →
|
|
│ EntryDetails13 → EntryTransaction14 →
|
|
│ TransactionReferences6.EndToEndId)
|
|
└── Cargo.toml — direct dep on iso20022-common (nested camt
|
|
types live there, not in the camt module)
|
|
|
|
crates/op-graph/ (additions)
|
|
├── src/graph.rs — vtypes::STATEMENT_LINE, RECONCILIATION_TASK;
|
|
│ etypes::RECONCILES, TASK_ABOUT
|
|
├── src/reconciliation_store.rs — NEW:
|
|
│ GraphReconciliationStore impls
|
|
│ op_reconciliation::ReconciliationStore.
|
|
│ Deterministic UUIDv5 for statement_line vertex
|
|
│ ids (re-record reuses the same vertex).
|
|
└── Cargo.toml — direct dep on op-reconciliation
|
|
```
|
|
|
|
### Vertex / edge schema additions
|
|
|
|
**Vertex types**: `statement_line`, `reconciliation_task`.
|
|
|
|
**Edge types**:
|
|
- `statement_line --reconciles--> ledger_tx` (matched line, reserved
|
|
for a future enrichment — see honest concerns)
|
|
- `reconciliation_task --about--> ledger_tx` or `statement_line`
|
|
(drawn when the discrepancy names one)
|
|
|
|
### Two-tier matching
|
|
|
|
```
|
|
Tier 1 (strong, deterministic):
|
|
index txs by external_id → on match, compare amount + status
|
|
├── currency+amount differ → AmountMismatch
|
|
├── amount ok, status != Posted → StatusMismatch
|
|
└── all good → matched
|
|
|
|
Tier 2 (heuristic):
|
|
unmatched lines → search unused txs for
|
|
same currency + same amount magnitude + |posted - effective| ≤ tol
|
|
├── hit → fuzzy_matched
|
|
└── miss → UnmatchedStatement
|
|
|
|
Leftover ledger txs in the window → UnmatchedLedger
|
|
```
|
|
|
|
Defaults: tier-2 tolerance = 86_400s (24h — banks routinely book
|
|
across cut-offs and weekends). Operator-tunable via
|
|
`Reconciler::with_fuzzy_tolerance_secs`.
|
|
|
|
## Key design decisions
|
|
|
|
### 1. Pluggable `ReconciliationSource`, not a hardcoded parser
|
|
|
|
The trait — `iter_lines(&self) -> Box<dyn Iterator<Item =
|
|
Result<StatementLine>> + '_>` — mirrors the pluggable-driver idiom of
|
|
`LedgerStore` / `WebhookStore`. Sync iterator (matches the rest of
|
|
the stack; async wrappers live downstream). **Lazy**: a malformed
|
|
line surfaces per-item, not at construction, so a 50,000-line CAMT
|
|
can be processed without fully materializing — and an operator's
|
|
bespoke CSV is a `~100`-LOC `impl ReconciliationSource` away.
|
|
|
|
### 2. Caller-selected ledger window
|
|
|
|
`Reconciler::reconcile(source, ledger_txs: &[Transaction])` takes the
|
|
ledger window as an explicit slice. We deliberately **don't** impose
|
|
a `list_transactions` method on `op_ledger::LedgerStore` — production
|
|
stores have millions of rows, and an unbounded scan is a footgun. The
|
|
caller (who knows their indexing strategy: date column, ledger,
|
|
batch) assembles the slice and hands it in. This is the honest, scope-
|
|
respecting boundary.
|
|
|
|
### 3. ISO 20022 traversal lives behind the `op-iso20022` facade
|
|
|
|
`op-reconciliation` does not depend on `open-payments-iso20022-camt`
|
|
or `iso20022-common`. The deep ISO 20022 walk — `BankToCustomerStatementV12`
|
|
→ `Stmt[]` → `Ntry[]` → `NtryDtls[]` → `TxDtls[]` → `Refs.EndToEndId`
|
|
— is encapsulated in `op-iso20022::statement::Message::camt053_entries()`
|
|
which produces a flat `StatementEntry`. Downstream sources consume
|
|
`StatementEntry`. If a future Phase 16 wants to support `camt.054`,
|
|
it adds `camt054_entries()` to the same module; the source layer is
|
|
unchanged.
|
|
|
|
### 4. `LineDirection` is intentionally not `op_ledger::Direction`
|
|
|
|
Ledger direction is a double-entry primitive (which side of which
|
|
account). Statement direction is the bank's plain-English "money came
|
|
in" / "money went out". Conflating the two is a classic reconciliation
|
|
bug — a credit on the bank statement is *not* the same thing as a
|
|
credit-side ledger entry — so they are distinct types and the matcher
|
|
never mixes them.
|
|
|
|
### 5. Discrepancies have deterministic `task_id`s
|
|
|
|
`Discrepancy::task_descriptor().task_id` is derived only from the
|
|
discrepancy's identifying fields (kind + source/tx id) — no clock, no
|
|
RNG. This is what makes `GraphReconciliationStore::record_report`
|
|
idempotent under repeated invocation: a nightly job that re-records
|
|
the same unresolved discrepancies creates no new tasks, just touches
|
|
the existing ones. The integration test
|
|
`reconciliation_tasks_wire_into_the_shared_ledger_graph` exercises
|
|
this end-to-end.
|
|
|
|
### 6. `ReconciliationStore` trait in `op-reconciliation`, impl in
|
|
`op-graph`
|
|
|
|
Same license-hygiene seam as the ledger and webhook stores: the
|
|
Apache-2.0-only operator can supply their own task store (Postgres,
|
|
Linear API, whatever) and never link the MPL-2.0 graph backend.
|
|
`op-graph::GraphReconciliationStore` is one implementation; it just
|
|
happens to be the one that wires tasks into the shared graph so an
|
|
operator can traverse from a ledger account to its open tasks in one
|
|
hop.
|
|
|
|
### 7. Webhook payload schema is *policy, not protocol*
|
|
|
|
`SETTLEMENT_EVENT_TYPE = "psp.settlement.confirmed"` plus the
|
|
`SettlementPayload` JSON shape (`source_id`, `external_id?`,
|
|
`amount_minor`, `currency`, `direction`, `posted_at_unix_secs?`) is
|
|
the reference contract OpenPay ships. An operator whose PSP emits a
|
|
different shape ships a different `ReconciliationSource`. The matcher
|
|
and engine never see JSON — they consume `StatementLine`s.
|
|
|
|
### 8. Aborting on a malformed line, not folding it into the report
|
|
|
|
A parse failure aborts the run rather than being silently dropped:
|
|
under-reporting discrepancies because we couldn't decode a line would
|
|
be a silent integrity bug. Document this in `Reconciler::reconcile`'s
|
|
`# Errors`. The webhook integration test
|
|
`malformed_settlement_payload_aborts_the_run` enforces it.
|
|
|
|
## What this crate does NOT do
|
|
|
|
- **No currency conversion / FX.** A line in EUR against a USD tx is
|
|
an `AmountMismatch`, not an FX calculation.
|
|
- **No optimal bipartite matching.** v1 is a deterministic two-tier
|
|
join, O(lines + txs). Optimal matching (Hungarian, etc.) is a
|
|
legitimate future request — it would change the matcher behind the
|
|
same `MatchOutcome` API.
|
|
- **No clock.** The window is caller-supplied; sources timestamp
|
|
their own lines. Replay is fully deterministic.
|
|
- **No auto-resolution.** We detect and classify discrepancies;
|
|
booking the correcting ledger entry is an operator decision (and
|
|
in most jurisdictions, a regulatorily significant one).
|
|
- **No `camt.054` source in v1.** The trait makes it trivial to add
|
|
— same `Ntry` shape — but it's a deliberate follow-on rather than
|
|
speculative scaffolding.
|
|
|
|
## Baseline detour
|
|
|
|
Phase 15 started with a substantial detour that has to be recorded
|
|
honestly: when the phase began the workspace did not build. **10 of
|
|
14 crates had compile errors** (~92 total), `cargo test` couldn't
|
|
run, and `cargo clippy --pedantic` was 270 lints deep. Prior phase
|
|
docs had test/LOC counts that the code had never observed.
|
|
|
|
Greening the baseline as part of Phase 15 cost roughly:
|
|
|
|
1. **Workspace `Cargo.toml`** — `uuid` lacked `v4` and `v5`; `time`
|
|
lacked `serde`/`formatting`/`parsing`/`macros`;
|
|
`open-payments-fednow` was pinned to a version (`=1.0.10`) that
|
|
was never published (the companion crate lagged at `1.0.9`).
|
|
2. **`op-iso20022`** — referenced sub-crate modules
|
|
(`open_payments_iso20022_pacs::...`) without depending on them;
|
|
the umbrella crate doesn't re-export submodules. Added direct
|
|
deps for each family. The pinned message versions had also drifted
|
|
(`pacs.008.001.08` → `.001.12`, etc.); bumped the variants to
|
|
match what `=1.0.10` actually exposes.
|
|
3. **Rust 2024 edition** — `#[no_mangle]` is now `#[unsafe(no_mangle)]`
|
|
across both FFI crates.
|
|
4. **`swift-bridge` 0.1.59 quirks** — its `IR` parser rejects `///`
|
|
doc comments and string-form `associated_to`; needs explicit
|
|
`swift_repr = "struct"` on shared structs; can't infer `&self`
|
|
across multiple opaque types in one `extern` block. Restructured
|
|
the bridge into one `extern "Rust"` block per opaque type with
|
|
typed receivers.
|
|
5. **`indradb` `AllVertexQuery.count()`** needs `CountQueryExt` in
|
|
scope (changed between 4.x and 5.0).
|
|
6. **Feature unification** — `op-vault` enables `op-core/pci-scope`,
|
|
which surfaces `PaymentMethod::RawPan` workspace-wide.
|
|
`op-orchestrator`'s exhaustive matches had to grow a `RawPan` arm
|
|
(or be `_`-wildcarded).
|
|
7. **Six genuine pre-existing logic bugs surfaced once tests ran**:
|
|
- `op-emv` TLV `children()` used the parent vertex's offset as the
|
|
base for nested entries instead of the value-bytes offset; added
|
|
`TlvRef::value_offset` and used it everywhere children iterate.
|
|
- `op-orchestrator` `FakeAcquirer` discarded the stored error and
|
|
substituted `Transport(...)`, masking `PspRejected` test
|
|
scenarios. (Required adding `Clone, PartialEq, Eq` to
|
|
`op_rails_card::Error`.)
|
|
- PIX client emitted `clearing_system: "BRSPB"` (the on-wire code)
|
|
for the internal model, but the `PixProfile` validator requires
|
|
`"ISPB"` (the Bacen participant-id scheme). The two crates had
|
|
never been compiled together.
|
|
- `op-webhook` retry test asserted `next_attempt_at_unix_secs >
|
|
1000` with `FixedJitter(0)` — full-jitter's valid lower bound is
|
|
0, so the assertion contradicted the deterministic jitter
|
|
choice. Replaced with the correct invariant: `>= now &&
|
|
<= now + max_delay`.
|
|
- `op-wasm` tests on the host hit the `JsValue`-error path, which
|
|
panics with "cannot convert to `JsValue` outside of the Wasm
|
|
target". Re-asserted the underlying validation in
|
|
`op_vault::CardData::new` on host; kept the wasm wrapper
|
|
assertion under `cfg(target_arch = "wasm32")`.
|
|
- `op-emv` `encoded_len_long_form` test stuffed 256 bytes of
|
|
`0xAA` filler into a *constructed* `0x6F` tag, then expected
|
|
the parser to accept it. The parser correctly rejects nested-
|
|
TLV junk inside a constructed tag; the test should have used a
|
|
primitive tag (now uses `0x5A`).
|
|
|
|
End state of the detour:
|
|
|
|
- `cargo build --workspace --all-targets`: **0 warnings, 0 errors**
|
|
- `cargo test --workspace`: **794 passing** (pre-Phase-15 baseline)
|
|
- `cargo clippy --workspace --all-targets`: **0 warnings**
|
|
|
|
Documented so a future Claude (or a future reader of `docs/`) can
|
|
distinguish "Phase 15 added 25 tests" from "Phase 15 also turned an
|
|
unverified narrative codebase into a building one."
|
|
|
|
## Bugs caught during construction (Phase 15 proper)
|
|
|
|
1. **Garbled doc on `Reconciler::reconcile`.** First draft of the doc
|
|
comment said "Lines that fail to parse are surfaced as
|
|
`UnmatchedStatement` is *not* how we handle them — instead a
|
|
parse failure aborts the run". Rewrote into two clean
|
|
paragraphs.
|
|
|
|
2. **`#[derive(Debug)]` on `Reconciler`.** Forgotten initially; the
|
|
engine test used `Reconciler::new(100, 50).unwrap_err()` which
|
|
requires the `Ok` type to be `Debug`.
|
|
|
|
3. **`cargo fix` over-removed `Money` / `Currency` / `HttpResponse`
|
|
imports** in lib code that were only used inside `#[cfg(test)]`
|
|
submodules. cargo fix runs against the default-config (non-test)
|
|
build and doesn't see test usages. Resolution: place test-only
|
|
imports inside the test `mod` rather than at module level so
|
|
cargo fix's view matches the lib view.
|
|
|
|
4. **`uuid` lacked `v5`.** Needed for deterministic statement-line
|
|
vertex ids in `GraphReconciliationStore` (so re-recording reuses
|
|
the same vertex). Added `v5` to the workspace `uuid` features
|
|
list.
|
|
|
|
5. **`open_payments_iso20022_camt::camt_053_001_12::ReportEntry14`
|
|
doesn't exist.** Nested camt types live in
|
|
`iso20022_common::common::*`; the camt module merely `use`s them
|
|
(not `pub use`). Added `iso20022-common = "=1.0.10"` as a direct
|
|
`op-iso20022` dependency and referenced types from
|
|
`iso20022_common::common`.
|
|
|
|
6. **`swift-bridge 0.1.59` rejected `///` doc comments inside the
|
|
bridge module** (it parses them as unrecognized `#[doc=...]`
|
|
attributes). Surfaced during baseline greening but worth noting:
|
|
the rule is "no attributes — not even doc — on shared
|
|
enums/structs inside `mod ffi`."
|
|
|
|
## Honest concerns going into Phase 16+
|
|
|
|
- **Tier-2 heuristic is best-effort.** Same currency + same amount
|
|
+ posted-within-tolerance picks the **first** matching tx. If the
|
|
same merchant posts two identical $19.99 sales 30 minutes apart
|
|
and the bank reports them out of order, the heuristic match is
|
|
technically wrong but the report is still correct (both pair up).
|
|
An optimal bipartite assignment would never pick the wrong pair,
|
|
but its non-determinism (under tie-breaks) is its own audit
|
|
hazard. Documented and deferred.
|
|
|
|
- **Multi-currency transactions can't be valued by a single line.**
|
|
An FX transaction whose debit side spans two currencies returns
|
|
`None` from `tx_settled_amount`, and any line matching it on
|
|
external id surfaces as `UnmatchedStatement` rather than a wrong
|
|
`AmountMismatch`. Correct but limited; real FX reconciliation is
|
|
per-currency and is a future enhancement.
|
|
|
|
- **CAMT.053 XML serde-round-trip is not yet conformance-tested.**
|
|
We added `Message::parse_camt053` but tested the flattening logic
|
|
via programmatically-built `BankToCustomerStatementV12` (matches
|
|
the codebase's existing convention — the pacs.008 conformance
|
|
tests are also substring-only). A real `vectors/camt053_*.xml`
|
|
with serde round-trip is a follow-on. The risk is that some
|
|
optional ISO 20022 field name our serde derives don't expect
|
|
could break decode for an upstream-sourced statement; the
|
|
flatten function itself is solid.
|
|
|
|
- **`StatementLine::reconciles --> ledger_tx` edges are not
|
|
emitted yet.** The schema reserves them; `Reconciler` currently
|
|
exposes only the discrepancy list, not the matched-pairs list.
|
|
Adding a `matches: Vec<(SourceId, TransactionId)>` to
|
|
`MatchOutcome` and persisting them is straightforward and lands
|
|
in Phase 16 or wherever an operator wants to traverse "which
|
|
statement line settled this tx."
|
|
|
|
- **Webhook payload schema lock-in.** `SETTLEMENT_EVENT_TYPE +
|
|
SettlementPayload` is policy. Any operator whose PSP emits a
|
|
different shape ships their own source — the abstraction is
|
|
correct but the reference schema isn't a standard. Worth
|
|
documenting clearly in any operator-facing docs we add later.
|
|
|
|
- **No bipartite or graph-aware matching across attempts.** When a
|
|
ledger transaction has been retried (so the original auth voided,
|
|
a new one posted), reconciling against the statement is ambiguous.
|
|
v1 matches on external id; production deployments often track
|
|
attempt history and the matcher should be able to consult it. The
|
|
trait surface is the right place to extend; the matcher itself
|
|
doesn't need to change.
|
|
|
|
## Cumulative state
|
|
|
|
(Numbers updated to reflect what `cargo test --workspace` actually
|
|
reports today, after the baseline detour. Earlier "tests/LOC" values
|
|
in phase docs were not observed.)
|
|
|
|
| Phase | Crate(s) | Tests (observed) | LOC (approx) |
|
|
|---|---|---:|---:|
|
|
| 1 | op-core | 19 | ~600 |
|
|
| 2 | op-iso20022 (pre-camt053) | 39 | ~1,400 |
|
|
| 3 | op-emv | 51 | ~1,800 |
|
|
| 4 | op-rails-card | 46 | ~2,100 |
|
|
| 5 | op-rails-a2a | 66 | ~3,200 |
|
|
| 6 | op-fraud | 65 | ~2,400 |
|
|
| 7 | op-vault | 51 | ~2,600 |
|
|
| 8 | op-ffi-swift | 44 | ~2,700 |
|
|
| 9 | op-ffi-jni | 69 | ~2,950 |
|
|
| 10 | op-wasm | 35 | ~2,200 |
|
|
| 11 | op-orchestrator + examples | 90 | ~4,150 |
|
|
| 12 | op-ledger | 69 | ~2,540 |
|
|
| 13 | op-webhook | 106 | ~3,304 |
|
|
| 14 | op-graph | 74 | ~3,696 |
|
|
| **15** | **op-reconciliation (+ iso20022 statement view + op-graph store)** | **25** | **~1,916** |
|
|
| **Total** | | **819** | **~37,556** |
|
|
|
|
## What's next (Phase 16+ candidates)
|
|
|
|
- **`camt.053` XML serde round-trip** vector + the corresponding
|
|
conformance test (matches the `pacs.008` pattern in
|
|
`op-iso20022/tests/conformance.rs`).
|
|
- **`Camt054Source`** — same neutral statement view, intra-day
|
|
notifications.
|
|
- **Matched-pair graph enrichment** — emit
|
|
`statement_line --reconciles--> ledger_tx` edges from the matched
|
|
set, so an operator can traverse from a tx to the bank line that
|
|
settled it.
|
|
- **`op-router` integration** — when the orchestrator picks a rail,
|
|
consult the graph for recently-failed endpoints and
|
|
reconciliation-task density per rail, and avoid the noisy ones.
|
|
- **Persistent `IndraDB` backend** — wire up `rocksdb-datastore`
|
|
via `GraphHandle::new_persistent(path)`. Recorded in the
|
|
Phase 14 doc; still pending.
|
|
- **Bipartite optimal matching** — opt-in alternative matcher
|
|
behind the same `MatchOutcome` API.
|
|
- **OpenTelemetry trace propagation** — one trace id flowing
|
|
intent → ledger → webhook → reconciliation.
|
|
|
|
The thesis stands: pure-Rust, Apache-2.0 reference stack; the
|
|
ledger/bank discrepancy that used to live in spreadsheets is now a
|
|
typed value an operator can route into automation.
|