open-standards/openpay/docs/observability/README.md

OpenPay Observability

The bi-temporal append-only ledger is one of OpenPay's architectural moats: every state transition is a recorded event with both a valid-time (when the event happened in the real world) and a transaction-time (when the ledger learned of it). Inspecting that moat requires three things: a CLI for ad-hoc queries, OpenTelemetry traces over the live Payment<S> typestate, and a Grafana dashboard to watch the system from the operations side.

This directory holds the second and third.

Wiring up

1. OpenTelemetry collector

# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
exporters:
  prometheus:
    endpoint: 0.0.0.0:9464
service:
  pipelines:
    metrics:
      receivers:  [otlp]
      exporters:  [prometheus]
    traces:
      receivers:  [otlp]
      exporters:  [prometheus]

2. OpenPay process

Compile op-orchestrator with the telemetry feature and set OPENPAY_OTLP_ENDPOINT=http://localhost:4317 in the environment. op_orchestrator::telemetry::init_telemetry(...) wires the global subscriber at startup. Every Payment<S> state transition emits a span with these attributes:

Attribute	Type	Example
payment.id	string	pay_019e30baa...
payment.state.from	string	Authorized
payment.state.to	string	Captured
payment.rail	string	card · a2a · crypto
payment.amount.minor	int	1299
payment.amount.currency	string	USD

3. Prometheus + Grafana

Scrape the collector's prometheus exporter at :9464. Import grafana-dashboard.json into Grafana. The dashboard ships with three template variables: $rail, $currency, and the standard $time_range.

Example bi-temporal query

What was account acct_019e3...'s balance on May 4 as we knew it on May 5?

op ledger as-of \
  --valid 2026-05-04T00:00:00Z \
  --transaction 2026-05-05T00:00:00Z \
  --account acct_019e30baa9507793b467ac644636b3e2 \
  --human

Output (with --human):

account                   acct_019e30baa9507793b467ac644636b3e2
valid-time                2026-05-04T00:00:00Z
transaction-time          2026-05-05T00:00:00Z
balance                   12_450 minor USD
entries-cited             487

Strip --human to get the same content as structured JSON for scripting and reconciliation tooling.

What the dashboard shows

• State-transition rate — transitions per second by (from, to) pair. Spikes on Authorized → Captured are the healthy steady state; * → Refunded rate is the dispute / refund pulse.
• In-flight by state — current count of payments in each typestate. Long tails in Authorized indicate captures that aren't landing.
• Refund vs capture ratio — short-window and long-window. Spikes beyond a configurable threshold can be wired into the alert manager.
• Rail split — card vs A2A vs crypto, by volume and by count.
• Latency per transition — p50 / p95 / p99 per state transition, bucketed by rail.
• Error rate per state — counts of Err(...) returns from each state-transition function, bucketed by rail and error class.

Recompute history

The bi-temporal property of the ledger is the real magic: when an old reconciliation finds a discrepancy in last week's data, recording the correction does not rewrite history. It appends a new entry with valid-time = last week, transaction-time = now. The CLI query above will return the correction when asked from "now" forward and the original (uncorrected) state when asked from before "now". The dashboard's data source does the same.