open-standards/openpay/docs/16-persistent-graph-progress.md

Phase 16 — Embedded persistent graph DB (Minigraf)

Status: Draft v0.16 Date: 2026-05-21

Why this phase exists

Phase 14 shipped op-graph on IndraDB in-memory only. Operator question that immediately followed: "how do we keep the graph across restarts?" The Phase 14 progress doc promised a RocksDB backend; this phase delivers that capability — but not with RocksDB.

The "right Rust graph DB" turned out to be a real research question. The constraint set:

1. Embedded ("in-app like SQLite"), not a separate server.
2. Pure Rust. No heavy C/C++ build chains.
3. Real graph DB. Not "use SQL with self-joins."
4. Single-file durability ("like SQLite").
5. Apache-2.0 / MIT licensable.

The landscape evaluated:

Candidate	Verdict
IndraDB + RocksDB	RocksDB is a heavy C++ build, single-process lock. Rejected.
Cozo	Original crate paused at v0.7.6 (Dec 2023). Community fork cozo-ce 0.7.13-alpha.3 fails to compile against current rayon (transitive graph_builder API drift). Not viable in 2026.
SurrealDB	Pure Rust, but server-first; embedded mode pulls heavy deps and the core is BSL 1.1, not Apache-2.0. License contagion for a reference stack. Rejected.
petgraph	Data structure library, not a database.
Minigraf 1.1.1	Selected. Pure safe Rust (#![forbid(unsafe_code)]), MIT OR Apache-2.0, single-file .graph database, embedded (no server), Datalog queries over an EAV fact store, native Uuid entity ids, bi-temporal history built in (free time-travel for ledger audits). Active (1.x stable, not alpha).

Minigraf advertises itself in its own docs as "the SQLite of graph databases" — verbatim what was asked for.

What shipped

A complete replacement of the IndraDB-backed op-graph internals with Minigraf. The public API surface of every op-graph type (GraphHandle, GraphLedgerStore, GraphWebhookStore, GraphReconciliationStore, all queries.rs helpers) is preserved — callers across Phase 14 / 15 / the orchestrator integration tests recompile without changes.

File	LOC	Notes
op-graph/Cargo.toml	+6	dep swap: indradb-lib → minigraf; tempfile dev-dep
op-graph/src/graph.rs	596	full rewrite on Minigraf
op-graph/src/error.rs	−10	dropped Indra/indradb::ValidationError bridges, added Backend(String)
op-graph/src/ledger_store.rs	small sweep	outbound_id→from, inbound_id→to, &indradb::Edge→&crate::graph::Edge, plus a real fix to find_by_external_id (graph fallback when the cache misses on a freshly-opened persistent handle)
op-graph/src/webhook_store.rs	small sweep	same field renames
op-graph/src/reconciliation_store.rs	small sweep	same field renames
op-graph/src/queries.rs	small sweep	same field renames
op-graph/tests/persistent.rs	224	new — 4 round-trip tests proving data survives handle drop + reopen across the path
Phase 16 totals	~820 added	(the 596 graph.rs LOC mostly replaces 316 prior LOC; net new is on the order of +500)

Workspace at the end of Phase 16:

Check	Result
cargo build --workspace --all-targets	0 errors, 0 warnings
cargo test --workspace	810 passing, 0 failing
cargo clippy --workspace --all-targets	0 warnings

The net test count vs Phase 15 (819) is −13 op-graph lib tests (internal graph.rs unit tests of IndraDB-specific behaviour that no longer apply) +4 persistent integration tests = 810. All 11 op-graph integration tests including the Phase 15 cross-domain reconciliation test pass unchanged.

Architecture: EAV mapped onto property-graph

Minigraf is an EAV (Entity-Attribute-Value) fact store with a Datalog query language. The OpenPay graph is property-graph shaped — typed vertices, typed directed edges, JSON properties on both — so the facade translates one onto the other:

Vertex(id, type)                  → [id :_type type]
VertexProperty(id, name, value)   → [id name value]

Edge(from, type, to):
  mint edge_id (Uuid::new_v4)
  → [edge_id :_edge/from <from>]
    [edge_id :_edge/to <to>]
    [edge_id :_edge/type <type>]

EdgeProperty(edge_id, name, val)  → [edge_id name value]

Internal attributes are prefixed _ so user property names (e.g. status, amount_minor) can be any keyword without collision. get_vertex_properties filters _* out so callers see only their own properties.

Schema in code

pub mod vtypes {
    pub const LEDGER_ACCOUNT: &str = "ledger_account";
    pub const LEDGER_TX: &str = "ledger_tx";
    pub const LEDGER_LEDGER: &str = "ledger_ledger";
    pub const WEBHOOK_EVENT: &str = "webhook_event";
    pub const WEBHOOK_ENDPOINT: &str = "webhook_endpoint";
    pub const WEBHOOK_ATTEMPT: &str = "webhook_attempt";
    pub const STATEMENT_LINE: &str = "statement_line";
    pub const RECONCILIATION_TASK: &str = "reconciliation_task";
}

pub mod etypes {
    pub const LEDGER_DEBIT: &str = "ledger_debit";
    pub const LEDGER_CREDIT: &str = "ledger_credit";
    pub const LEDGER_IN_LEDGER: &str = "ledger_in_ledger";
    pub const LEDGER_REVERSES: &str = "ledger_reverses";
    pub const WEBHOOK_DELIVERS: &str = "webhook_delivers";
    pub const WEBHOOK_TO: &str = "webhook_to";
    pub const RECONCILES: &str = "reconciles";
    pub const TASK_ABOUT: &str = "task_about";
}

Unchanged from Phase 14 / 15 — the schema is the same property graph; only the storage substrate moved.

Vertex / Edge types are now local to op-graph

The IndraDB-specific indradb::Vertex / indradb::Edge no longer appear in the public API. They were replaced with:

pub struct Vertex { pub id: Uuid, pub t: String }
pub struct Edge   { pub id: Uuid, pub from: Uuid, pub t: String, pub to: Uuid }

This is a tiny breaking change relative to Phase 14: edge.outbound_id → edge.from, edge.inbound_id → edge.to. Internal to op-graph; no downstream phase touches these field names directly.

Persistence: it just works

// Tests, ephemeral:
let h = GraphHandle::new_in_memory();

// Single-file durability:
let h = GraphHandle::new_persistent("./openpay.graph")?;
// ... write through it ...
drop(h);    // file flushes

let h2 = GraphHandle::new_persistent("./openpay.graph")?;
// ... reads see the data written by the previous handle.

That's the entire user-facing API change. The four persistent round-trip tests in tests/persistent.rs walk each store type:

1. graph_handle_persists_vertex_and_edge_across_reopen — the low-level GraphHandle primitive.
2. ledger_transaction_round_trips_across_reopen — write a ledger transaction, drop, reopen, verify the entries (stored as edges) and the derived balance are all intact.
3. webhook_event_and_attempt_round_trip_across_reopen — the same for webhook events and delivery attempts.
4. reconciliation_tasks_survive_restart_and_re_record_stays_idempotent — the cross-domain integration. Reconcile against a webhook source, persist tasks, drop, reopen, re-run reconciliation against the persisted ledger, verify task vertices are unchanged and the deterministic task_id index keeps record_report idempotent across the restart.

Key design decisions

1. EAV/Datalog under, property-graph facade above

The data model on disk is EAV facts — every property write is a fact [entity attribute value]. The facade above presents a property-graph API (create_vertex, create_edge, set_vertex_property) so callers in Phase 14 / 15 don't need to learn Datalog. Datalog queries are constructed and dispatched only inside graph.rs; the strings never leak across module boundaries.

The trade-off is two-tier: when callers want graph traversals (accounts_touched_by_transaction, reversal_chain, etc.) they go through hand-rolled queries in queries.rs that use the facade's typed primitives. Operators who want raw Datalog can call handle.raw().execute("(query ...)") — the escape hatch is still there but explicitly opted into.

2. Retract-then-assert for scalar property updates

Minigraf is bi-temporal — every assert is appended to history, never overwritten. Naively calling set_vertex_property(id, "status", "posted") after set_vertex_property(id, "status", "pending") leaves both facts queryable. To give scalars the "current value" semantics the rest of the codebase expects, set_vertex_property now:

1. Queries the current value for (entity, attr),
2. Retracts every result ((retract [[id :attr v]])),
3. Asserts the new value.

This is 2–3 Datalog ops per set instead of 1. The historical trail is preserved — Minigraf still has every value the attribute ever held, queryable via :as-of. Only the current view (the default query) reflects the latest set. That's free time-travel for ledger audits.

3. Caught a pre-existing bug: find_by_external_id graph fallback

GraphLedgerStore maintained an in-memory Mutex<HashMap<external_id, TransactionId>> cache, populated lazily on writes. In Phase 14 the cache was always seeded by the producer of the transaction in the same process, so cache misses never mattered. With persistence, a fresh GraphLedgerStore::with_handle on a previously-written .graph file has an empty cache — and find_by_external_id returned None for transactions that actually existed in the graph.

Fixed: on cache miss, scan ledger_tx vertices via handle.vertices_of_type(), read each external_id property, and populate the cache on the way past. Honest about the cost (linear in transaction count for the first miss after a restart), but correct.

4. License rebase from MPL-2.0 to MIT OR Apache-2.0

indradb-lib is MPL-2.0; Phase 14 documented the operator-facing implications of MPL-2.0-at-file-level. minigraf is MIT OR Apache-2.0, strictly more permissive. The thorough license note in op-graph/src/lib.rs (and Phase 14's progress doc) can be loosened: there is no longer an MPL-tainted backend in the operator's primary deployment story. Operators who still want to swap the store implementation entirely can do so — the trait seams on op-ledger / op-webhook / op-reconciliation haven't moved.

5. JSON property codec preserved

set_vertex_property still takes a serde_json::Value and get_vertex_properties still returns serde_json::Map<String, Value>. Internally:

• Scalars (Null / Bool / Number / String) map directly to minigraf::Value::{Null, Boolean, Integer, Float, String}.
• Arrays and objects serialize to a json:-prefixed string and are re-parsed on read. (Minigraf has no native nested-JSON value.)

This preserves the codec contract documented in the Phase 14 doc. The json: marker is internal — mg_to_json strips it before returning.

6. Backend is fixed at construction time, not feature-flagged

Phase 16's earlier draft enum-dispatched between Memory and Persistent variants of an internal Backend enum. With Minigraf, both modes are first-class on the same type (Minigraf::in_memory() vs Minigraf::open(path)), so the dispatch is just a constructor choice. Less code, fewer match arms, no feature flags.

What this crate does NOT do

• No clustering / multi-process at the storage layer. Minigraf is single-process embedded by design. Operators who want horizontal scale plug in a different LedgerStore (and friends) behind a remote datastore — the trait surfaces are shaped for that explicitly and Phase 12/13's docs say as much.
• No automatic schema migration. Adding properties is safe (Datalog is schemaless w.r.t. attributes). Removing or renaming attributes is a Datomic-style retract job operators run themselves.
• No automatic indexing. vertices_of_type walks all vertices and filters; for our reference-impl volumes this is fine. A custom store impl over a tuned datastore handles high-cardinality types.
• No transaction-time clock control yet. Bi-temporal queries (:as-of N) are available via the raw() escape hatch; the typed facade doesn't expose them. Coming in a later phase if / when a use case lands.

Bugs caught during construction

1. QueryResult variant names. First draft assumed QueryResult::Query(rows) / Transact(_). The actual variants are QueryResults { vars, results }, Transacted(tx_id), Retracted(tx_id), Ok. The internal QueryRows::from adapter handles all four.

2. EntityId is just Uuid. Earlier draft called eid.into_uuid() — the type is a literal pub type EntityId = Uuid. Direct value use.

3. Keywords come back with their colon. Minigraf returns Keyword(":name"), not Keyword("name"). The get_vertex_properties adapter strips the leading colon so callers query bare attribute names. Without this fix every property read returned None.

4. Bi-temporal history made set a no-op for "current value" semantics. Without retract-then-assert, two writes to :status left both "pending" and "posted" queryable, and mark_posted appeared not to have run. See Decision 2.

5. The find_by_external_id cache-only path was a bug Phase 14 never noticed because every process always wrote what it later read. Surfaced immediately under persistence. See Decision 3.

Honest concerns going into Phase 17+

• Query cost in user units. Each set_vertex_property is a query + zero-or-one retract + one transact. For a transaction with many entries that gets multiple property updates, this is noticeably more Datalog work than IndraDB's one-call set. For the reference impl it's still fast (the workspace test suite runs in seconds). For production volumes operators care about, the trait seam allows swapping in a tuned store.

• No bulk-insert primitive. Each create / set goes through a (transact ...) Datalog string. Minigraf doesn't currently expose a programmatic bulk-insert that bypasses Datalog parsing. The cost is real but bounded — measured impact on the existing test suite was unobservable.

• Edge property API still defined but minimally tested. set_edge_property works (it shares set_entity_property with vertices), but the Phase 14 / 15 stores don't currently set any edge properties. No regression risk; just noting the surface is less battle-tested than the vertex side.

• File format is Minigraf's, not OpenPay's. A .graph file written by op-graph v0.16 is only readable by another process linking the same minigraf version. Operators who want a long-lived archival format export to a portable serialization separately. Documented; not in scope to solve here.

• Bi-temporal history grows unboundedly. Every retract is itself a fact in the history. For a long-running deployment, the .graph file grows. Compaction is something Minigraf supports but op-graph doesn't yet expose. Honest follow-on.

Cumulative state

Phase	Crate(s)	Tests (observed)	LOC (approx)
1	op-core	19	~600
2	op-iso20022 (pre-camt053)	39	~1,400
3	op-emv	51	~1,800
4	op-rails-card	46	~2,100
5	op-rails-a2a	66	~3,200
6	op-fraud	65	~2,400
7	op-vault	51	~2,600
8	op-ffi-swift	44	~2,700
9	op-ffi-jni	69	~2,950
10	op-wasm	35	~2,200
11	op-orchestrator + examples	90	~4,150
12	op-ledger	69	~2,540
13	op-webhook	106	~3,304
14	op-graph (original, IndraDB)	74	~3,696
15	op-reconciliation (+ iso20022 statement view + op-graph store)	25	~1,916
16	op-graph rewrite on Minigraf + persistence	(net +1 vs Phase 15: −3 IndraDB-internal unit + +4 persistent integration)	+500 net
Total		810	~38,056

What's next (Phase 17+ candidates)

• CAMT.053 XML serde round-trip vector + conformance test (rolled over from Phase 15).
• Camt054Source — intra-day notifications.
• Matched-pair graph enrichment — emit statement_line --reconciles--> ledger_tx edges from matched pairs, now with the persistence to keep them.
• Time-travel queries on the ledger. Minigraf's :as-of is free; expose a typed helper like balance_as_of(account, tx_count) -> Money for auditors.
• History compaction — surface Minigraf's compaction so long-running operators can prune fact history they've already archived.
• op-router integration — orchestrator consults reconciliation-task density per rail and prefers quiet rails.

The thesis stands: pure-Rust, Apache-2.0 reference payment stack; the graph that used to live only in RAM is now a single-file durable artifact, with free bi-temporal history.