open-standards/openpay/docs/14-graph-progress.md

Phase 14 — op-graph graph-backed stores

Status: Draft v0.14 Date: 2026-05-18

Why a graph database, not Postgres

OpenPay's data is graph-shaped. A relational schema flattens that shape into joins; a graph schema preserves it. Two examples:

1. A ledger transaction is a hyperedge connecting 2+ accounts. "Which accounts did this transaction touch?" in relational SQL is SELECT a.* FROM entries e JOIN accounts a ON e.account_id = a.id WHERE e.transaction_id = $1. In a graph it's one traversal of out-edges. The graph version is also faster on wide ledgers because the edges sit next to the transaction vertex on disk.

2. Reversal chains are subgraphs. Tracing tx → reverses → tx → reverses → tx in relational SQL is a recursive CTE. In a graph it's a four-line loop. Same with fraud-graph queries ("which accounts share a device fingerprint via attempted chargebacks"), which become first-class operations.

We chose IndraDB because:

• Pure Rust, embedded MemoryDatastore available out of the box,
• TAO-inspired (Facebook's graph datastore) so query semantics scale to the size of a real ledger,
• Mature (~10 years, 2.4k stars), latest release 5.0.0 (2025-08-16),
• Pluggable datastores — operators can swap in RocksDB for persistence later without touching our code.

License compatibility

This is the section I want to be honest about up front: IndraDB is MPL-2.0, and OpenPay is Apache-2.0. They are compatible in the way that matters for OpenPay operators: a downstream deployment that links against IndraDB from an Apache-2.0 binary does not have to relicense the rest of its code. The MPL-2.0 obligation is at the file level: any modifications to IndraDB's own source files must remain MPL-2.0. We never modify IndraDB; we only link.

This is the same composition model that ships in Firefox-embedded applications across the industry. Operators who want a single- license deployment can substitute a different LedgerStore / WebhookStore implementation — the trait surfaces in op-ledger and op-webhook exist precisely for that.

The license note is reproduced in crates/op-graph/src/lib.rs and called out explicitly in this doc so no operator is surprised later.

What shipped

A new crate crates/op-graph plus an integration test suite that exercises a single shared GraphHandle across both ledger and webhook stores. Total: 74 tests (64 unit + 10 integration), ~3,696 LOC. Cumulative across all 14 phases: ~870 tests, ~35,640 LOC.

Module	Unit tests	LOC
lib.rs	—	117
error.rs	—	105
graph.rs (typed facade over IndraDB)	13	453
ledger_store.rs (GraphLedgerStore)	20	1,135
webhook_store.rs (GraphWebhookStore)	20	1,056
queries.rs (read-side helpers)	11	380
tests/integration.rs	10	450
Phase 14 total	74	3,696

Architecture

crates/op-graph/
├── Cargo.toml         — deps: op-core, op-ledger, op-webhook,
│                       indradb-lib = "=5.0.0" pinned tight
├── src/
│   ├── lib.rs         — module roots + schema doc + license note
│   ├── error.rs       — 9-variant Error with From<indradb::Error>,
│   │                   From<indradb::ValidationError>,
│   │                   From<serde_json::Error>, plus bridges to
│   │                   op_ledger::Error and op_webhook::Error
│   ├── graph.rs       — GraphHandle: typed facade over
│   │                   Database<MemoryDatastore>. vtypes / etypes
│   │                   string constant modules.
│   ├── ledger_store.rs— GraphLedgerStore: implements
│   │                   op_ledger::LedgerStore. JSON property
│   │                   codecs; HashMap<String, TransactionId>
│   │                   external_id cache for idempotency.
│   ├── webhook_store.rs— GraphWebhookStore: implements
│   │                   op_webhook::WebhookStore. Base64 codec for
│   │                   binary payload + secret. CSV codec for
│   │                   event_filters. Per-event-type endpoint
│   │                   index.
│   └── queries.rs     — 4 read-side helpers
└── tests/
    └── integration.rs — 10 cross-domain tests

Vertex / edge schema

Vertex types: ledger_account, ledger_tx, ledger_ledger, webhook_event, webhook_endpoint, webhook_attempt.

Edge types:

• ledger_debit: ledger_tx → ledger_account (amount_minor, currency_code, currency_exponent)
• ledger_credit: ledger_tx → ledger_account (same props)
• ledger_in_ledger: ledger_tx or ledger_account → ledger_ledger
• ledger_reverses: ledger_tx → ledger_tx (the original)
• webhook_delivers: webhook_event → webhook_attempt
• webhook_to: webhook_attempt → webhook_endpoint

Why edge-per-entry instead of entries-as-properties

Two options for storing transaction entries:

1. Property blob: serialize the entire Vec<Entry> as JSON and stash on the tx vertex.
2. Edge per entry: each entry becomes a typed edge from the tx vertex to the account vertex.

Option 1 is simpler but makes "which accounts did this tx touch?" an O(N) scan of every tx in the database. Option 2 makes it a single out-edge traversal — which is the whole reason we picked a graph database. So edge-per-entry it is.

Cost: a tx with k entries is k+1 vertices' worth of edges (plus the ledger_in_ledger edge to the parent ledger). Acceptable.

Key design decisions

1. GraphHandle is the seam

The crate exposes a thin facade rather than letting callers pass raw Database<MemoryDatastore> references around. Rationale: the OpenPay schema (vtypes, etypes, property names) lives in one place, and operators who want to swap to a different graph database (e.g. via a custom datastore implementation) can do so by replacing one file.

GraphHandle::raw() is the escape hatch for advanced queries the facade doesn't model. Calling raw() opts out of stability.

2. Shared GraphHandle across stores

GraphLedgerStore::with_handle(handle.clone()) and GraphWebhookStore::with_handle(handle.clone()) let both stores write to one graph. This unlocks the cross-domain queries: "this webhook event was emitted because this ledger transaction was posted — show me both in one read." Demonstrated in integration test #9.

3. Idempotency cache mirrors InMemoryLedgerStore

GraphLedgerStore keeps a Mutex<HashMap<String, TransactionId>> seeded lazily on write. It's not authoritative — the graph is — but it makes find_by_external_id O(1) instead of an all-vertices scan. Production stores plugging in a persistent datastore would replace this with an indexed property query (IndraDB supports with_property_equal_to plus property indexing); the in-memory facade doesn't bother because the cache is fast enough.

4. JSON property codecs are explicit

We don't serde_json::to_value(&account)? and stash the whole struct as a blob. Every field is encoded explicitly with a known property name and known type (currency_code as string, amount_minor as i64, effective_at_unix_secs as u64). Reasons:

• Future schema evolution: when we add a property, we don't have to deserialize old blobs.
• Query-friendly: VertexWithPropertyValueQuery works against individual properties; a blob property is opaque.
• Audit-friendly: an operator browsing the graph (via the indradb-server CLI) sees readable property names.

The cost is more code in the codec helpers, but they're trivial and test-covered.

5. Local base64 + filter-CSV codecs, not extra deps

The webhook payload and endpoint secret are Vec<u8>. JSON can't carry raw bytes, so we encode. We wrote a ~60-line base64 codec rather than adding the base64 crate — same dependency hygiene as the local hex codec in op-webhook. RFC 4648 test vectors verified in base64_known_vectors.

The event-filter list is encoded as a CSV inside one property (escaping , as \,). It could've been a JSON array, but keeping it textual lets IndraDB indexing work on the full string (future optimization) and avoids the JSON-property-of-arrays edge cases.

6. Sealed Error::Invariant

When the graph is in an unexpected state (e.g. a tx vertex with no debit/credit edges, or a property of the wrong JSON shape), we raise Error::Invariant(String). Callers shouldn't normally see these — they'd indicate either schema drift (someone wrote through the raw IndraDB API bypassing our facade) or a bug in our codec. The variant exists so we can be explicit instead of panicking.

7. License-friendly module boundary

op-graph depends on indradb-lib, but op-ledger and op-webhook do NOT — the trait surfaces they expose are storage-agnostic. Operators who want to ship under a strict Apache-2.0 hygiene policy can use op-ledger and op-webhook with their own (Apache-2.0 or MIT) LedgerStore / WebhookStore implementation, simply not depending on op-graph.

What this crate does NOT do

• No graph query language. No Cypher / GQL / Gremlin. IndraDB itself doesn't expose those; we expose typed Rust helpers (the four functions in queries.rs).
• No persistence by default. MemoryDatastore only. IndraDB supports persistent backends via its rocksdb-datastore feature (we don't enable it; operators choosing persistence opt in by depending on indradb-lib with that feature and passing their own Database<RocksdbDatastore> to a custom store).
• No cross-store transactionality. Writing a ledger transaction and a webhook event is two store calls. They're both backed by IndraDB so they share the underlying mutex, but there's no rollback if the second one fails.
• No streaming subscriptions. Operators poll or run periodic queries.
• No fraud-graph queries yet. The schema supports them (everything is in one graph), but the typed helpers in queries.rs cover only the four ledger + webhook traversals. Fraud queries land in Phase 16 or later.

Bugs caught during construction

1. MPL-2.0 vs Apache-2.0 compatibility surprise. I initially assumed IndraDB was Apache-2.0 or MIT. Searching turned up MPL-2.0. License analysis added to the architectural doc to set operator expectations correctly.

2. IndraDB API drift between major versions. Earlier docs showed outbound(limit: u32) -> PipeEdgeQuery. The 5.0.0 surface is outbound() -> ValidationResult<PipeQuery> with .limit() and .t() as builder chains. Pinned to =5.0.0 to avoid this surprise upstream.

3. Orphan vertex leak on put_attempt. First draft of GraphWebhookStore::put_attempt created the attempt vertex before verifying the event and endpoint vertices existed. If validation failed, an orphan vertex was left behind. Fixed to validate first, create after.

4. Status property is not the tx's lifecycle state. When mark_posted is called, we have to update the status property on the tx vertex, not just call Transaction::post on the in-memory copy. First draft updated the local struct but didn't write back; balance() then read the old status and miscounted. Fixed by writing the new status property after the state-machine method succeeds.

5. Currency constants vs arbitrary codes. My currency_from_props(code, exponent) first tries the seven curated constants (USD/EUR/BRL/INR/GBP/JPY/CNY) for the fast-path, then falls through to Currency::try_new for arbitrary codes. Without the fast-path the constants reconstructed via try_new would be != to the original despite encoding the same currency, because the comparison sees them as semantically equivalent but structurally distinct values. (Actually they would be equal — Currency derives PartialEq on its raw byte fields. But the curated path is faster and avoids the validation work, so we keep it.)

6. Direction-as-property vs direction-as-edge-type. I initially considered storing direction as a property on a single ledger_entry edge type. Decided against: making debit and credit separate edge types lets us filter by direction in a single query (out_edges(tx, "ledger_debit")) rather than out-edges + property filter. Trade-off: two edge types instead of one. Worth it.

7. SpecificVertexQuery::single(uuid).outbound()?.t(ident) composition. outbound() returns ValidationResult<PipeQuery>; t() consumes PipeQuery and returns PipeQuery. Took two tries to get the ordering right — the chain looks naturally ?.t(...) but I initially wrote it as .t(ident)? (calling t on a Result, which doesn't compile).

Honest concerns going into Phase 15+

• balance() is O(edges-on-account), not O(1). For high-throughput accounts (e.g. a corporate cash account with millions of transactions) this is slow. A real production deployment would cache the running balance and update it on every entry. Out of scope for the reference impl.

• list_due_retries does a full-vertex scan. IndraDB doesn't (yet) give us a typed-vertex iterator at the embedded API level — we have to walk AllVertexQuery and filter by type and property. For a webhook system with tens of thousands of attempts this is fine; for millions it isn't. Fix: when IndraDB exposes typed scans, or when we plug in RocksDB and use property indexing.

• No cycle protection in reversal_chain beyond a HashSet. If an operator manually wires t1 → reverses → t2 and t2 → reverses → t1 (which shouldn't happen but the graph doesn't forbid), we detect the cycle and return what we've found so far. We don't error. Documented in the function comment.

• The license note is operator-facing. Anyone considering redistributing OpenPay needs to read it. We surface it in three places (lib.rs doc comment, Cargo.toml description, this progress doc) so no one is surprised.

Cumulative state

Phase	Tests	LOC
1 op-core	19	~600
2 op-iso20022	43	~1,400
3 op-emv	50	~1,800
4 op-rails-card	46	~2,100
5 op-rails-a2a	73	~3,200
6 op-fraud	65	~2,400
7 op-vault	51	~2,600
8 op-ffi-swift	44	~2,700
9 op-ffi-jni	69	~2,950
10 op-wasm	71	~2,200
11 op-orchestrator + kiosk + e2e	90	~4,150
12 op-ledger	69	~2,540
13 op-webhook	106	~3,304
14 op-graph	74	~3,696
Total	~870	~35,640

What's next (Phase 15+ candidates)

• op-reconciliation — diff ledger vs PSP / bank statements; produces per-discrepancy resolution tasks. Natural follow-on now that we have a graph-backed ledger to query.
• Persistent IndraDB backend — wire up rocksdb-datastore via a GraphHandle::new_persistent(path) constructor. Mostly one file's worth of work.
• Fraud-graph queries — accounts_linked_via_chargeback, endpoints_sharing_secret_prefix, attempts_with_shared_ip. Two-hop traversals on the existing schema; no new edge types needed.
• op-router integration — when the orchestrator decides which rail to use, it can consult the graph for recently-failed endpoints and avoid them.
• OpenTelemetry trace propagation — one trace id flowing intent → ledger → webhook → consumer.
• GraphQL adapter — expose the typed queries.rs helpers as a GraphQL schema so operator dashboards can poll without building Rust.

The thesis stands: pure-Rust, Apache-2.0 reference stack with a license-compatible MPL-2.0 graph backend chosen because OpenPay's data is graph-shaped.