70ecd6caa0
Transaction Science open standards monorepo. Each subdirectory is a self-contained reference protocol with its own README and licence. - openpay/ Apache-2.0 payment-acceptance stack (Rust, 26 crates) - smart-byte/ CC-BY-4.0 content-addressed value carrier (spec) - eoc/ CC-BY-4.0 energy-optimised compute (spec) The wire format and the right to fork are public. The optional hosted services are the offer; the protocols are not.
358 lines
16 KiB
Markdown
358 lines
16 KiB
Markdown
# Phase 13 — `op-webhook` outbound delivery
|
|
|
|
**Status**: Draft v0.13
|
|
**Date**: 2026-05-18
|
|
|
|
## What shipped
|
|
|
|
A new crate `crates/op-webhook` plus an integration test suite that
|
|
demonstrates a ledger-event-driven fanout against a mock HTTP
|
|
transport. **Total: 106 tests (96 unit + 10 integration), ~3,304
|
|
LOC.**
|
|
|
|
The crate is the **fanout edge** of the stack: ledger and
|
|
orchestrator events emerge here as signed HTTPS POSTs to merchant
|
|
endpoints, with retry, dead-letter, replay, and auto-disable
|
|
handling. The wire format is byte-for-byte compatible with
|
|
Stripe's so any merchant who has already integrated a Stripe
|
|
webhook can verify ours with the same code.
|
|
|
|
## Verified ground truth
|
|
|
|
Researched live (May 2026 sources) before implementation:
|
|
|
|
| Claim | Source |
|
|
|---|---|
|
|
| Exponential backoff with **full jitter** is the industry consensus for webhook retry | AWS "Exponential Backoff and Jitter" (2015); Stripe, GitHub, Shopify "built-in exponential backoff with jitter" (Hooklistener 2026) |
|
|
| **72-hour total retry window** | Stripe ("retries failures with exponential backoff for up to 72 hours") |
|
|
| **24-hour window then auto-disable** | Razorpay ("retry the delivery in exponential backoff policy for 24 hours") |
|
|
| **Auto-disable after ~10 consecutive failures** | Hookdeck Outpost, dev.to webhook-system guide |
|
|
| **Stripe header format**: `Stripe-Signature: t={unix_secs},v1={hex_hmac_sha256(secret, "{ts}.{body}")}` | Stripe docs, Hooklistener 2026, multiple independent implementations confirm |
|
|
| **5-minute default timestamp tolerance window** | Stripe SDKs, Hooklistener 2026 |
|
|
| **Constant-time comparison** (`subtle::ConstantTimeEq`) required to prevent timing attacks | Stripe security guide, OWASP, multiple references |
|
|
| **Retry on**: 5xx, 408, 425, 429, transport failures | Hookdeck, integrate.io 2026 |
|
|
| **Don't retry on**: 2xx, 3xx, and 4xx other than 408/425/429 | Hookdeck, docsfordevs |
|
|
| `hmac` crate API: `Hmac::<Sha256>::new_from_slice(secret)`, `mac.update(bytes)`, `mac.finalize().into_bytes()` | docs.rs/hmac |
|
|
|
|
## Architecture
|
|
|
|
```
|
|
crates/op-webhook/
|
|
├── Cargo.toml — deps: op-core, serde, thiserror, uuid, subtle, sha2, hmac
|
|
├── src/
|
|
│ ├── lib.rs — module roots + pub re-exports
|
|
│ ├── error.rs — 11-variant Error enum
|
|
│ ├── hexutil.rs — tiny in-crate hex encoder (no `hex` dep)
|
|
│ ├── signing.rs — Stripe-compatible HMAC-SHA256 + verify with constant-time compare
|
|
│ ├── retry.rs — RetryPolicy trait, ExponentialBackoffPolicy, jitter source seam
|
|
│ ├── event.rs — WebhookEvent, DeliveryAttempt, DeliveryStatus
|
|
│ ├── endpoint.rs — Endpoint, EndpointId, EndpointStatus, filter matching
|
|
│ ├── transport.rs — HttpTransport trait, MockTransport
|
|
│ ├── store.rs — WebhookStore trait + InMemoryWebhookStore
|
|
│ └── dispatcher.rs — WebhookDispatcher: dispatch / replay / process_due_retries
|
|
└── tests/
|
|
└── integration.rs — 10 tests demonstrating ledger-event-driven fanout
|
|
```
|
|
|
|
### Data flow
|
|
|
|
```
|
|
┌──────────────────────────┐
|
|
│ WebhookEvent │
|
|
│ (opaque Vec<u8> body) │
|
|
└────────────┬─────────────┘
|
|
│
|
|
┌────────────────┴──────────────────┐
|
|
│ WebhookDispatcher │
|
|
│ │
|
|
│ for each matching endpoint: │
|
|
│ 1. compute HMAC over │
|
|
│ "{now}.{body}" │
|
|
│ 2. build OpenPay-Signature │
|
|
│ header │
|
|
│ 3. transport.send() │
|
|
│ 4. classify response: │
|
|
│ 2xx → Delivered │
|
|
│ 5xx/408/425/429 → Retry │
|
|
│ other 4xx → DeadLetter │
|
|
│ transport err → Retry │
|
|
│ 5. record DeliveryAttempt │
|
|
│ 6. update endpoint │
|
|
│ consecutive_failures │
|
|
│ 7. auto-disable if threshold │
|
|
│ crossed │
|
|
└───────────────────────────────────┘
|
|
```
|
|
|
|
## Key design decisions
|
|
|
|
### 1. Stripe-compatible signing scheme
|
|
|
|
```
|
|
OpenPay-Signature: t=1700000000,v1=4f4c...
|
|
```
|
|
|
|
The signed payload is `"{ts}.{body}"`; the HMAC key is the
|
|
endpoint's secret. This means an existing Stripe-compatible
|
|
verifier (`stripe.Webhook.constructEvent`, or any community
|
|
implementation) verifies our payloads with only the header NAME
|
|
changed. Operators get a 30-second integration.
|
|
|
|
We accept multiple `v1=` entries in a single header, which is
|
|
Stripe's mechanism for key rotation. We don't currently emit
|
|
multiple — that's a future extension.
|
|
|
|
### 2. Exponential backoff with full jitter, NOT equal jitter
|
|
|
|
Three jitter strategies are documented in the literature:
|
|
|
|
- **Full**: `delay = rand(0, base * 2^n)`. Most spread, best at
|
|
avoiding thundering herd. **Our default.**
|
|
- **Equal**: `delay = base * 2^(n-1) + rand(0, base * 2^(n-1))`.
|
|
Guarantees some minimum delay.
|
|
- **Decorrelated**: `delay = rand(base, prev * 3)`. Adaptive.
|
|
|
|
Full jitter matches AWS's 2015 recommendation and what
|
|
Stripe/GitHub/Shopify appear to do based on observed behavior. The
|
|
[`RetryPolicy`] trait lets operators substitute a different
|
|
strategy without touching the dispatcher.
|
|
|
|
### 3. Pluggable HTTP transport, no real client in this crate
|
|
|
|
We deliberately don't depend on `reqwest`, `hyper`, or `ureq`.
|
|
Operators inject one through the [`HttpTransport`] trait. This
|
|
keeps the dep footprint minimal and decouples the async/sync story
|
|
from the dispatcher.
|
|
|
|
The reference [`MockTransport`] is the test fixture: programmable
|
|
per-call responses, request capture for assertions.
|
|
|
|
### 4. Pluggable jitter RNG for deterministic tests
|
|
|
|
`RetryPolicy::next_delay_secs` calls a [`JitterRng`] trait
|
|
([`SystemJitter`] in production, [`FixedJitter`] in tests). This
|
|
keeps tests reproducible without mocking the system clock for the
|
|
jitter alone.
|
|
|
|
### 5. Auto-disable after N consecutive failures
|
|
|
|
Per the Razorpay / dev.to community patterns. After
|
|
`disable_after_consecutive_failures` consecutive failures across
|
|
ANY events for one endpoint, the endpoint's status flips to
|
|
[`EndpointStatus::AutoDisabled`] and `list_active_endpoints_for`
|
|
filters it out. Operators must explicitly re-enable. Any
|
|
successful delivery resets the counter to 0.
|
|
|
|
### 6. Explicit replay, never silent resurrection
|
|
|
|
After auto-disable or a 4xx dead-letter, events stay in the store
|
|
indefinitely. Operators trigger
|
|
[`WebhookDispatcher::replay(event_id, endpoint_id)`] manually after
|
|
fixing the root cause. The dispatcher never silently retries a
|
|
dead-lettered attempt.
|
|
|
|
**Defensive default**: even `replay()` honors `EndpointStatus`'s
|
|
`is_blocking()`. Operators wanting to replay against a disabled
|
|
endpoint must first re-enable it. This avoids "I clicked replay
|
|
and immediately tripped the auto-disable I was trying to recover
|
|
from" footguns.
|
|
|
|
### 7. The dispatcher is sync; async is operator-level
|
|
|
|
Matches the rest of the stack. Operators run N dispatcher threads
|
|
(one per worker pool) or wrap in `tokio::task::spawn_blocking`.
|
|
|
|
### 8. Tiny local hex encoder, not the `hex` crate
|
|
|
|
The `hex` crate is 8KB compiled but it's another supply-chain
|
|
surface to audit. We need only encode/decode; 50 lines of code,
|
|
7 unit tests, done.
|
|
|
|
### 9. Constant-time signature comparison
|
|
|
|
`subtle::ConstantTimeEq::ct_eq` is used in `verify_signature`. A
|
|
naive `==` on hex strings would short-circuit on the first
|
|
mismatching byte; an attacker who can measure verification latency
|
|
could extract the correct signature one byte at a time. `ct_eq`
|
|
always processes the full input.
|
|
|
|
### 10. Idempotency at the consumer, NOT at the dispatcher
|
|
|
|
Webhook delivery is *at-least-once* by design. Consumers must
|
|
dedupe by `WebhookEvent.id` (sent as the `OpenPay-Event-Id`
|
|
header). The dispatcher doesn't try to enforce
|
|
exactly-once-delivery (an impossible property over an unreliable
|
|
network).
|
|
|
|
## What this crate does NOT do
|
|
|
|
- **No real HTTP client.** Operators inject one.
|
|
- **No async runtime.** Sync trait surfaces; operators wrap.
|
|
- **No durable queue.** `InMemoryWebhookStore` is single-process.
|
|
Production deployments swap in Postgres/Redis-backed stores.
|
|
- **No CloudEvents formatting.** The payload is opaque; operators
|
|
encode however they like.
|
|
- **No key rotation flow.** A single secret per endpoint. The
|
|
signature scheme already supports multiple `v1=` entries during
|
|
rotation — extension is documented as future work.
|
|
- **No PII redaction.** Operators redact upstream.
|
|
- **No bulk signing.** One event, one HMAC computation per
|
|
endpoint. Bulk operations are out of scope.
|
|
|
|
## Composition with the rest of the stack
|
|
|
|
The 10-test integration suite at `tests/integration.rs` shows the
|
|
operator pattern: a synthetic `ledger.transaction.posted` payload
|
|
flows through the dispatcher to a merchant endpoint, with the
|
|
merchant's `verify_signature` call (acting as receiver) succeeding
|
|
when the payload is intact and failing when tampered.
|
|
|
|
```
|
|
┌──────────────────┐ ┌───────────────────┐ ┌──────────────────┐
|
|
│ op-ledger │ posted │ op-webhook │ HTTPS │ merchant ERP │
|
|
│ transaction ────┼─event───► dispatcher ────POST──► verify_sig() │
|
|
│ │ │ │ │ process() │
|
|
└──────────────────┘ └───────────────────┘ └──────────────────┘
|
|
```
|
|
|
|
Notably **`op-webhook` has no compile-time dependency on
|
|
`op-ledger`**. The payload type is `Vec<u8>`; the event type is a
|
|
free-form string. Decoupled, by design.
|
|
|
|
## Bugs caught during construction
|
|
|
|
1. **Function names starting with digits.** Initial draft had
|
|
`fn 5xx_response_schedules_retry`, `fn 4xx_response_dead_letters_immediately`,
|
|
`fn 429_response_schedules_retry` — all illegal Rust
|
|
identifiers. Renamed to `five_hundred_three_response_schedules_retry`,
|
|
`four_oh_four_response_dead_letters_immediately`,
|
|
`four_twenty_nine_response_schedules_retry`.
|
|
|
|
2. **Filter glob semantics.** Initial test used filter `"payment.*"`
|
|
expecting it to match `"payment.authorized"`, but the filter
|
|
implementation is **literal compare** (with `*` as the
|
|
"everything" wildcard, not a glob meta-character). Either:
|
|
(a) implement true glob matching, (b) change the test to use
|
|
the documented literal/wildcard semantics. Picked (b) because
|
|
adding glob matching brings either a new dep (`glob` crate) or
|
|
~80 lines of pattern code with its own test surface, and the
|
|
intended OpenPay event taxonomy is short enough (low single
|
|
digits) that operators can list filters explicitly. Documented
|
|
in the [`Endpoint::matches`] doc comment.
|
|
|
|
3. **Workspace dep alignment.** `subtle` was already in
|
|
`[workspace.dependencies]` at version 2.6; my crate manifest
|
|
originally pinned its own `subtle = "2"`. Aligned to
|
|
`subtle = { workspace = true }` for consistency.
|
|
|
|
4. **Tiny hex crate temptation.** First draft was going to add the
|
|
`hex` crate for `hex::encode`/`hex::decode`. Caught and replaced
|
|
with a 50-line local implementation (`hexutil.rs`) plus 7 unit
|
|
tests. Saves a supply-chain audit and a transitive
|
|
`serde + serde_derive` chain on some `hex` versions.
|
|
|
|
5. **Audit trail in `process_due_retries`.** First draft only
|
|
created NEW attempts on retry without marking the old one
|
|
terminal. `list_due_retries` would have re-picked the OLD
|
|
attempt forever. Fixed by marking the prior attempt as
|
|
`DeliveryStatus::Failed` (the "superseded" sentinel) before
|
|
creating the new attempt.
|
|
|
|
6. **Replay vs disabled endpoint.** First draft of `replay()` did
|
|
NOT check `EndpointStatus::is_blocking()`, so operators could
|
|
accidentally re-trigger an auto-disable storm by replaying
|
|
against a still-disabled endpoint. Fixed to honor the blocking
|
|
check; operators must re-enable first. Documented + tested.
|
|
|
|
7. **Excerpt length on response bodies.** Bodies returned by a
|
|
misconfigured receiver could be MB-sized HTML pages. The
|
|
dispatcher truncates to 512 bytes for the audit record
|
|
(`RESPONSE_BODY_EXCERPT_BYTES`). Tested in
|
|
`excerpt_truncates_large_body`.
|
|
|
|
## Test count
|
|
|
|
| Module | Unit tests |
|
|
|---|---|
|
|
| `dispatcher.rs` | 18 |
|
|
| `endpoint.rs` | 11 |
|
|
| `event.rs` | 6 |
|
|
| `hexutil.rs` | 7 |
|
|
| `retry.rs` | 17 |
|
|
| `signing.rs` | 22 |
|
|
| `store.rs` | 10 |
|
|
| `transport.rs` | 5 |
|
|
| **Unit total** | **96** |
|
|
| `tests/integration.rs` | **10** |
|
|
| **Phase 13 total** | **106** |
|
|
|
|
### Integration test summary
|
|
|
|
1. `fanout_happy_path_signature_verifies_on_receiver_side` — full
|
|
round-trip: dispatcher signs, mock transport captures, receiver
|
|
verifies with the same secret.
|
|
2. `modified_body_fails_signature_verification` — tampering check.
|
|
3. `old_timestamp_outside_tolerance_is_rejected` — replay
|
|
protection.
|
|
4. `endpoint_auto_disables_after_threshold_failures` —
|
|
AutoDisabled flips and subsequent dispatches are skipped.
|
|
5. `fanout_multiple_endpoints_each_get_signed_independently` —
|
|
per-endpoint secrets are isolated; one endpoint's secret can
|
|
never verify another's signature.
|
|
6. `retry_succeeds_via_process_due_retries` — 5xx → retry → 200.
|
|
7. `replay_works_after_operator_reenables_endpoint` — recovery
|
|
workflow.
|
|
8. `four_oh_four_dead_letters_without_retry` — 4xx fails fast.
|
|
9. `signature_header_is_parseable_by_stripe_style_consumers` —
|
|
on-the-wire compatibility check.
|
|
10. `audit_trail_records_every_attempt` — every attempt persists
|
|
for forensics.
|
|
|
|
## Cumulative state
|
|
|
|
| Phase | Tests | LOC |
|
|
|---|---|---|
|
|
| 1 op-core | 19 | ~600 |
|
|
| 2 op-iso20022 | 43 | ~1,400 |
|
|
| 3 op-emv | 50 | ~1,800 |
|
|
| 4 op-rails-card | 46 | ~2,100 |
|
|
| 5 op-rails-a2a | 73 | ~3,200 |
|
|
| 6 op-fraud | 65 | ~2,400 |
|
|
| 7 op-vault | 51 | ~2,600 |
|
|
| 8 op-ffi-swift | 44 | ~2,700 |
|
|
| 9 op-ffi-jni | 69 | ~2,950 |
|
|
| 10 op-wasm | 71 | ~2,200 |
|
|
| 11 op-orchestrator + kiosk + e2e | 90 | ~4,150 |
|
|
| 12 op-ledger | 69 | ~2,540 |
|
|
| **13 op-webhook** | **106** | **~3,304** |
|
|
| **Total** | **~796** | **~31,944** |
|
|
|
|
## What's next
|
|
|
|
Phase 14+ candidates (not committed):
|
|
|
|
- **Postgres-backed `WebhookStore` and `LedgerStore`** —
|
|
production-ready persistent backends; would graduate the
|
|
in-memory references from "fine for kiosks" to "fine for
|
|
Series-A merchants".
|
|
- **`op-reconciliation`** — diff ledger vs. PSP / bank statements;
|
|
the webhook receiver above is the natural ingestion path for the
|
|
reconciliation pipeline.
|
|
- **TigerBeetle-backed `LedgerStore`** — high-throughput backend.
|
|
- **OpenTelemetry trace propagation** — single trace id flowing
|
|
from orchestrator intent → ledger metadata → webhook attempt →
|
|
consumer.
|
|
- **CloudEvents v1.0 envelope** — optional output adapter so
|
|
events are CNCF-compatible by default.
|
|
- **Key rotation flow** — emit both old and new `v1=` signatures
|
|
during a configurable cutover window.
|
|
- **Endpoint health metrics** — p50/p95/p99 latency,
|
|
delivery-success rate, drained-DLQ counts; pluggable
|
|
`WebhookObserver` trait.
|
|
- **Async transport adapter** — `tokio::task::spawn_blocking`
|
|
wrapper that lets tokio-native callers integrate without
|
|
blocking the runtime.
|
|
|
|
The thesis stands: pure-Rust, Apache-2.0, no platform lock-in, every
|
|
event signed with a key the merchant controls and verifiable with
|
|
the same code they already use for Stripe.
|