Payment Gateway Architecture: Idempotency Keys, Ledger Design, and Exactly-Once Money Movement

Introduction

In most software, a duplicate request is annoying. In payments, it is a lawsuit.

Payment gateways sit at the worst possible intersection: distributed systems, external bank networks with unpredictable latency, and users who hammer "Pay" when the spinner hangs. Without deliberate architecture, you will double-charge, lose track of funds, or report balances that do not reconcile.

This guide covers the three pillars of production payment architecture: idempotency, ledger design, and reconciliation.

Section 1: Idempotency Keys End-to-End

Every payment intent must carry a client-generated Idempotency-Key (UUID v4).

Server behavior

Receive request with key,
Check idempotency store (Redis or Postgres with unique constraint),
If key exists and completed → return cached response,
If key exists and in-flight → return 409 or wait with timeout,
If new → process, persist result, return response.

Scope

Idempotency keys must be unique per merchant + operation type + client. Document TTL (typically 24–72 hours) and behavior after expiry.

Webhooks too

Payment providers send duplicate webhooks. Treat provider_event_id as an idempotency key in your webhook handler—never apply the same settlement twice.

Section 2: Ledger Design

Never update a balance column directly as your source of truth. Use an append-only ledger.

Double-entry model

Every money movement creates at least two entries:

Debit:  customer_wallet    $100
Credit: settlement_account $100

Balance is derived: SUM(credits) - SUM(debits) per account. Corrections are new entries—not updates to old rows.

Transaction states

initiated → pending → settled | failed | reversed

External provider state and internal ledger state must be explicitly linked. A payment_id maps to one ledger transaction group.

Immutability

Ledger rows are never deleted or edited. Reversals create compensating entries with reference to the original transaction_id.

Section 3: Reconciliation

Daily (or hourly at scale), compare:

internal ledger totals,
payment provider settlement reports,
bank statement deposits.

Discrepancies trigger investigation queues—not silent adjustments.

Common drift causes

late-arriving webhooks,
partial captures and refunds,
currency conversion rounding,
provider fees recorded in separate fee accounts.

Section 4: Failure Handling

Scenario	Response
Timeout after provider accepted	Reconcile via provider API before retry
Duplicate client retry	Return original result via idempotency key
Provider 5xx	Mark pending, async poll, never guess success
Partial refund	New ledger entries, link to parent payment

Conclusion

Payment architecture is distributed systems engineering with legal consequences. Idempotency keys, append-only ledgers, and daily reconciliation are not optional—they are the minimum viable trust layer.