# Phase 14 — `op-graph` graph-backed stores **Status**: Draft v0.14 **Date**: 2026-05-18 ## Why a graph database, not Postgres OpenPay's data is graph-shaped. A relational schema flattens that shape into joins; a graph schema preserves it. Two examples: 1. **A ledger transaction is a hyperedge connecting 2+ accounts.** "Which accounts did this transaction touch?" in relational SQL is `SELECT a.* FROM entries e JOIN accounts a ON e.account_id = a.id WHERE e.transaction_id = $1`. In a graph it's one traversal of out-edges. The graph version is also faster on wide ledgers because the edges sit next to the transaction vertex on disk. 2. **Reversal chains are subgraphs.** Tracing `tx → reverses → tx → reverses → tx` in relational SQL is a recursive CTE. In a graph it's a four-line loop. Same with fraud-graph queries ("which accounts share a device fingerprint via attempted chargebacks"), which become first-class operations. We chose **IndraDB** because: - Pure Rust, embedded `MemoryDatastore` available out of the box, - TAO-inspired (Facebook's graph datastore) so query semantics scale to the size of a real ledger, - Mature (~10 years, 2.4k stars), latest release 5.0.0 (2025-08-16), - Pluggable datastores — operators can swap in RocksDB for persistence later without touching our code. ## License compatibility This is the section I want to be honest about up front: **IndraDB is MPL-2.0**, and OpenPay is **Apache-2.0**. They are compatible in the way that matters for OpenPay operators: a downstream deployment that **links** against IndraDB from an Apache-2.0 binary does not have to relicense the rest of its code. The MPL-2.0 obligation is at the **file** level: any modifications to IndraDB's own source files must remain MPL-2.0. We never modify IndraDB; we only link. This is the same composition model that ships in Firefox-embedded applications across the industry. Operators who want a single- license deployment can substitute a different `LedgerStore` / `WebhookStore` implementation — the trait surfaces in `op-ledger` and `op-webhook` exist precisely for that. The license note is reproduced in `crates/op-graph/src/lib.rs` and called out explicitly in this doc so no operator is surprised later. ## What shipped A new crate `crates/op-graph` plus an integration test suite that exercises a single shared `GraphHandle` across both ledger and webhook stores. **Total: 74 tests (64 unit + 10 integration), ~3,696 LOC.** Cumulative across all 14 phases: ~870 tests, ~35,640 LOC. | Module | Unit tests | LOC | |---|---|---| | `lib.rs` | — | 117 | | `error.rs` | — | 105 | | `graph.rs` (typed facade over IndraDB) | 13 | 453 | | `ledger_store.rs` (`GraphLedgerStore`) | 20 | 1,135 | | `webhook_store.rs` (`GraphWebhookStore`) | 20 | 1,056 | | `queries.rs` (read-side helpers) | 11 | 380 | | `tests/integration.rs` | 10 | 450 | | **Phase 14 total** | **74** | **3,696** | ## Architecture ``` crates/op-graph/ ├── Cargo.toml — deps: op-core, op-ledger, op-webhook, │ indradb-lib = "=5.0.0" pinned tight ├── src/ │ ├── lib.rs — module roots + schema doc + license note │ ├── error.rs — 9-variant Error with From, │ │ From, │ │ From, plus bridges to │ │ op_ledger::Error and op_webhook::Error │ ├── graph.rs — GraphHandle: typed facade over │ │ Database. vtypes / etypes │ │ string constant modules. │ ├── ledger_store.rs— GraphLedgerStore: implements │ │ op_ledger::LedgerStore. JSON property │ │ codecs; HashMap │ │ external_id cache for idempotency. │ ├── webhook_store.rs— GraphWebhookStore: implements │ │ op_webhook::WebhookStore. Base64 codec for │ │ binary payload + secret. CSV codec for │ │ event_filters. Per-event-type endpoint │ │ index. │ └── queries.rs — 4 read-side helpers └── tests/ └── integration.rs — 10 cross-domain tests ``` ### Vertex / edge schema **Vertex types**: `ledger_account`, `ledger_tx`, `ledger_ledger`, `webhook_event`, `webhook_endpoint`, `webhook_attempt`. **Edge types**: - `ledger_debit`: `ledger_tx` → `ledger_account` (`amount_minor`, `currency_code`, `currency_exponent`) - `ledger_credit`: `ledger_tx` → `ledger_account` (same props) - `ledger_in_ledger`: `ledger_tx` or `ledger_account` → `ledger_ledger` - `ledger_reverses`: `ledger_tx` → `ledger_tx` (the original) - `webhook_delivers`: `webhook_event` → `webhook_attempt` - `webhook_to`: `webhook_attempt` → `webhook_endpoint` ### Why edge-per-entry instead of entries-as-properties Two options for storing transaction entries: 1. **Property blob**: serialize the entire `Vec` as JSON and stash on the tx vertex. 2. **Edge per entry**: each entry becomes a typed edge from the tx vertex to the account vertex. Option 1 is simpler but makes "which accounts did this tx touch?" an O(N) scan of every tx in the database. Option 2 makes it a single out-edge traversal — which is the whole reason we picked a graph database. So edge-per-entry it is. Cost: a tx with k entries is k+1 vertices' worth of edges (plus the `ledger_in_ledger` edge to the parent ledger). Acceptable. ## Key design decisions ### 1. `GraphHandle` is the seam The crate exposes a thin facade rather than letting callers pass raw `Database` references around. Rationale: the OpenPay schema (vtypes, etypes, property names) lives in one place, and operators who want to swap to a different graph database (e.g. via a custom datastore implementation) can do so by replacing one file. `GraphHandle::raw()` is the escape hatch for advanced queries the facade doesn't model. Calling `raw()` opts out of stability. ### 2. Shared `GraphHandle` across stores `GraphLedgerStore::with_handle(handle.clone())` and `GraphWebhookStore::with_handle(handle.clone())` let both stores write to **one** graph. This unlocks the cross-domain queries: "this webhook event was emitted because this ledger transaction was posted — show me both in one read." Demonstrated in integration test #9. ### 3. Idempotency cache mirrors `InMemoryLedgerStore` `GraphLedgerStore` keeps a `Mutex>` seeded lazily on write. It's not authoritative — the graph is — but it makes `find_by_external_id` O(1) instead of an all-vertices scan. Production stores plugging in a persistent datastore would replace this with an indexed property query (IndraDB supports `with_property_equal_to` plus property indexing); the in-memory facade doesn't bother because the cache is fast enough. ### 4. JSON property codecs are explicit We don't `serde_json::to_value(&account)?` and stash the whole struct as a blob. Every field is encoded explicitly with a known property name and known type (`currency_code` as string, `amount_minor` as i64, `effective_at_unix_secs` as u64). Reasons: - **Future schema evolution**: when we add a property, we don't have to deserialize old blobs. - **Query-friendly**: `VertexWithPropertyValueQuery` works against individual properties; a blob property is opaque. - **Audit-friendly**: an operator browsing the graph (via the indradb-server CLI) sees readable property names. The cost is more code in the codec helpers, but they're trivial and test-covered. ### 5. Local base64 + filter-CSV codecs, not extra deps The webhook payload and endpoint secret are `Vec`. JSON can't carry raw bytes, so we encode. We wrote a ~60-line base64 codec rather than adding the `base64` crate — same dependency hygiene as the local hex codec in op-webhook. RFC 4648 test vectors verified in `base64_known_vectors`. The event-filter list is encoded as a CSV inside one property (escaping `,` as `\,`). It could've been a JSON array, but keeping it textual lets IndraDB indexing work on the full string (future optimization) and avoids the JSON-property-of-arrays edge cases. ### 6. Sealed `Error::Invariant` When the graph is in an unexpected state (e.g. a tx vertex with no debit/credit edges, or a property of the wrong JSON shape), we raise `Error::Invariant(String)`. Callers shouldn't normally see these — they'd indicate either schema drift (someone wrote through the raw IndraDB API bypassing our facade) or a bug in our codec. The variant exists so we can be explicit instead of panicking. ### 7. License-friendly module boundary `op-graph` depends on `indradb-lib`, but `op-ledger` and `op-webhook` do NOT — the trait surfaces they expose are storage-agnostic. Operators who want to ship under a strict Apache-2.0 hygiene policy can use `op-ledger` and `op-webhook` with their own (Apache-2.0 or MIT) `LedgerStore` / `WebhookStore` implementation, simply not depending on `op-graph`. ## What this crate does NOT do - **No graph query language.** No Cypher / GQL / Gremlin. IndraDB itself doesn't expose those; we expose typed Rust helpers (the four functions in `queries.rs`). - **No persistence by default.** `MemoryDatastore` only. IndraDB supports persistent backends via its `rocksdb-datastore` feature (we don't enable it; operators choosing persistence opt in by depending on `indradb-lib` with that feature and passing their own `Database` to a custom store). - **No cross-store transactionality.** Writing a ledger transaction and a webhook event is two store calls. They're both backed by IndraDB so they share the underlying mutex, but there's no rollback if the second one fails. - **No streaming subscriptions.** Operators poll or run periodic queries. - **No fraud-graph queries yet.** The schema supports them (everything is in one graph), but the typed helpers in `queries.rs` cover only the four ledger + webhook traversals. Fraud queries land in Phase 16 or later. ## Bugs caught during construction 1. **MPL-2.0 vs Apache-2.0 compatibility surprise.** I initially assumed IndraDB was Apache-2.0 or MIT. Searching turned up MPL-2.0. License analysis added to the architectural doc to set operator expectations correctly. 2. **IndraDB API drift between major versions.** Earlier docs showed `outbound(limit: u32) -> PipeEdgeQuery`. The 5.0.0 surface is `outbound() -> ValidationResult` with `.limit()` and `.t()` as builder chains. Pinned to `=5.0.0` to avoid this surprise upstream. 3. **Orphan vertex leak on put_attempt.** First draft of `GraphWebhookStore::put_attempt` created the attempt vertex *before* verifying the event and endpoint vertices existed. If validation failed, an orphan vertex was left behind. Fixed to validate first, create after. 4. **Status property is not the tx's lifecycle state.** When `mark_posted` is called, we have to update the **status property** on the tx vertex, not just call `Transaction::post` on the in-memory copy. First draft updated the local struct but didn't write back; balance() then read the old status and miscounted. Fixed by writing the new status property after the state-machine method succeeds. 5. **`Currency` constants vs arbitrary codes.** My `currency_from_props(code, exponent)` first tries the seven curated constants (USD/EUR/BRL/INR/GBP/JPY/CNY) for the fast-path, then falls through to `Currency::try_new` for arbitrary codes. Without the fast-path the constants reconstructed via `try_new` would be `!=` to the original despite encoding the same currency, because the comparison sees them as semantically equivalent but structurally distinct values. (Actually they would be equal — `Currency` derives `PartialEq` on its raw byte fields. But the curated path is faster and avoids the validation work, so we keep it.) 6. **Direction-as-property vs direction-as-edge-type.** I initially considered storing `direction` as a property on a single `ledger_entry` edge type. Decided against: making debit and credit separate edge types lets us filter by direction in a single query (`out_edges(tx, "ledger_debit")`) rather than out-edges + property filter. Trade-off: two edge types instead of one. Worth it. 7. **`SpecificVertexQuery::single(uuid).outbound()?.t(ident)` composition.** `outbound()` returns `ValidationResult`; `t()` consumes `PipeQuery` and returns `PipeQuery`. Took two tries to get the ordering right — the chain looks naturally `?.t(...)` but I initially wrote it as `.t(ident)?` (calling `t` on a Result, which doesn't compile). ## Honest concerns going into Phase 15+ - **`balance()` is O(edges-on-account)**, not O(1). For high-throughput accounts (e.g. a corporate cash account with millions of transactions) this is slow. A real production deployment would cache the running balance and update it on every entry. Out of scope for the reference impl. - **`list_due_retries` does a full-vertex scan.** IndraDB doesn't (yet) give us a typed-vertex iterator at the embedded API level — we have to walk `AllVertexQuery` and filter by type and property. For a webhook system with tens of thousands of attempts this is fine; for millions it isn't. Fix: when IndraDB exposes typed scans, or when we plug in RocksDB and use property indexing. - **No cycle protection in `reversal_chain` beyond a `HashSet`.** If an operator manually wires `t1 → reverses → t2` and `t2 → reverses → t1` (which shouldn't happen but the graph doesn't forbid), we detect the cycle and return what we've found so far. We don't error. Documented in the function comment. - **The license note is operator-facing.** Anyone considering redistributing OpenPay needs to read it. We surface it in three places (lib.rs doc comment, Cargo.toml description, this progress doc) so no one is surprised. ## Cumulative state | Phase | Tests | LOC | |---|---|---| | 1 op-core | 19 | ~600 | | 2 op-iso20022 | 43 | ~1,400 | | 3 op-emv | 50 | ~1,800 | | 4 op-rails-card | 46 | ~2,100 | | 5 op-rails-a2a | 73 | ~3,200 | | 6 op-fraud | 65 | ~2,400 | | 7 op-vault | 51 | ~2,600 | | 8 op-ffi-swift | 44 | ~2,700 | | 9 op-ffi-jni | 69 | ~2,950 | | 10 op-wasm | 71 | ~2,200 | | 11 op-orchestrator + kiosk + e2e | 90 | ~4,150 | | 12 op-ledger | 69 | ~2,540 | | 13 op-webhook | 106 | ~3,304 | | **14 op-graph** | **74** | **~3,696** | | **Total** | **~870** | **~35,640** | ## What's next (Phase 15+ candidates) - **`op-reconciliation`** — diff ledger vs PSP / bank statements; produces per-discrepancy resolution tasks. Natural follow-on now that we have a graph-backed ledger to query. - **Persistent IndraDB backend** — wire up `rocksdb-datastore` via a `GraphHandle::new_persistent(path)` constructor. Mostly one file's worth of work. - **Fraud-graph queries** — `accounts_linked_via_chargeback`, `endpoints_sharing_secret_prefix`, `attempts_with_shared_ip`. Two-hop traversals on the existing schema; no new edge types needed. - **`op-router` integration** — when the orchestrator decides which rail to use, it can consult the graph for recently-failed endpoints and avoid them. - **OpenTelemetry trace propagation** — one trace id flowing intent → ledger → webhook → consumer. - **GraphQL adapter** — expose the typed `queries.rs` helpers as a GraphQL schema so operator dashboards can poll without building Rust. The thesis stands: pure-Rust, Apache-2.0 reference stack with a license-compatible MPL-2.0 graph backend chosen because OpenPay's data is graph-shaped.