Webhooks and Idempotency in Payments Systems

Treat every payment webhook as at-least-once and possibly out-of-order. Put a client-supplied idempotency key on each outbound request, dedupe inbound events by their event id, verify the HMAC signature with a timestamp tolerance, retry with exponential backoff, and reconcile final state from the payment provider rather than trusting delivery order.

That is the whole discipline in one paragraph. The rest of this piece explains why each part is non-negotiable and how to build it so a duplicate charge, a replayed event, or a webhook that lands before its own prerequisite never corrupts your ledger. This is the layer where correctness is existential, and it is the layer teams most often get subtly wrong.

Why are payment webhooks at-least-once instead of exactly-once?

Because exactly-once delivery is impossible over an unreliable network. A provider that sends an event and never sees your acknowledgement cannot know whether you processed it or the response was lost, so it retries. Retrying is the safe choice, which means your endpoint will sometimes see the same event twice.

Stripe states this plainly: “Webhook endpoints might occasionally receive the same event more than once,” and recommends you “guard against duplicated event receipts by logging the event IDs you’ve processed” (Stripe webhooks). Adyen frames the consumer’s job the same way: “accept webhooks that you receive with a 2xx HTTP status code, store the message, and process the contents” (Adyen webhooks).

At-least-once is not a defect to be tolerated. It is the contract. Once you accept that duplicates and reordering are normal traffic, the design follows: every handler must be idempotent, and final truth must come from the provider’s API, not from the sequence in which packets happened to arrive. If you are still choosing a provider, the delivery and retry semantics belong in that comparison — we cover them in Stripe vs. Adyen for a fintech.

What is an idempotency key and how does it work on the request side?

An idempotency key is a unique token the client attaches to a state-changing request so the server can recognize a retry of that exact request and return the original outcome instead of performing the operation twice. It protects against your own network timeouts: you fire a charge, the connection drops, and you cannot tell whether it succeeded.

The mechanics, using Stripe’s implementation as the reference, work like this (Stripe idempotent requests):

• The client generates the key. Stripe suggests “V4 UUIDs, or another random string with enough entropy to avoid collisions,” up to 255 characters. Generate it before the first attempt and reuse the same value for every retry of that logical operation.
• The server stores the first result. Stripe saves “the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails. Subsequent requests with the same key return the same result, including 500 errors.”
• Parameters are fingerprinted. The idempotency layer “compares incoming parameters to those of the original request and errors if they’re not the same,” which stops you from accidentally reusing a key across two genuinely different operations.
• There is a dedup window. Keys are retained for 24 hours; a key reused after the original is pruned generates a new request. Your retry logic must live inside that window.
• Scope is POST. Keys apply to state-changing methods; GET and DELETE ignore them.

The IETF has an expired draft, “The Idempotency-Key HTTP Header Field,” that generalizes this pattern to make “non-idempotent HTTP methods such as POST or PATCH fault-tolerant” (IETF draft). It never reached RFC status, but it documents the shared model most payment APIs already follow.

Storing the first response correctly

The subtle part is atomicity. Insert the idempotency key and the operation’s result in the same transaction that performs the side effect, or use the key as a unique constraint so a concurrent duplicate loses a race rather than double-executing. Save results only after execution begins, never on validation failures, so a malformed request does not poison the key for a corrected retry.

How do you consume webhooks idempotently?

Dedupe on the provider’s event id and make the handler an upsert. When an event arrives, check whether you have already recorded that id; if so, acknowledge and stop. If not, record the id and apply the effect in the same transaction. The Standard Webhooks spec is explicit that consumers should use the “webhook-id as an idempotency key to prevent accidentally processing the same webhook more than once” (Standard Webhooks).

Deduplication alone is not enough, because two different events can describe the same underlying object and arrive in the wrong order. The durable fix is a state machine per entity. Model the payment’s lifecycle explicitly — created → authorized → captured → refunded — and have each handler perform a guarded transition that ignores anything stale.

• A captured event that lands while the record is already refunded is dropped, not applied.
• A late authorized event for an already-captured payment is a no-op.
• Only forward transitions defined by the machine mutate state.

This makes ordering irrelevant: whatever sequence the events arrive in, the entity converges to the correct terminal state. Combine that with event-id dedup and your consumer is safe against both replays and reordering without any assumption about delivery order.

How do you verify webhook signatures and stop replay attacks?

Verify an HMAC signature on every inbound webhook before you act on it, and reject events whose signed timestamp falls outside a tolerance window. Signature verification proves the event came from your provider; the timestamp check stops an attacker from capturing a valid event and replaying it later.

Stripe is blunt about the stakes: “Without verification, an attacker could send fake webhook events to your endpoint to trigger actions like fulfilling orders, granting account access, or modifying records” (Stripe webhooks). The mechanism is HMAC-SHA256 over the raw payload using your endpoint’s signing secret, delivered in a Stripe-Signature header that also carries a timestamp. Because the timestamp is inside the signed payload, “an attacker can’t change the timestamp without invalidating the signature.” Stripe’s default replay tolerance is five minutes.

Standard Webhooks generalizes the same construction: the signed base string is msg_id.timestamp.payload, the id and timestamp “must not be user controlled,” verification must “verify the webhook-timestamp header has a timestamp that is within some allowable tolerance,” and comparisons must use a “constant time comparison function” to avoid timing side channels.

Verification checklist

1. Read the raw request body — sign and verify the exact bytes, never a re-serialized object.
2. Recompute the HMAC with your signing secret and compare in constant time.
3. Confirm the signed timestamp is within tolerance (five minutes is a common default).
4. Only then parse the payload and hand it to the idempotent handler.

Signature verification and reconciliation are the kind of controls a technical acquirer will probe; if you are heading into a raise, see the fintech website that passes diligence.

How should retries and backoff work?

Retries should be automatic, bounded, and spaced with exponential backoff so a struggling consumer is not hammered while it recovers. On the sending side, providers already do this — Stripe “attempts to deliver events to your destination for up to three days with an exponential back off in live mode” (Stripe webhooks). Your job is to make your endpoint safe to be retried against.

Two rules keep retries from becoming a second failure mode. First, acknowledge fast: “return a successful status code (2xx) prior to any complex logic that could cause a timeout.” Enqueue the verified event and return 200 immediately; do the heavy processing asynchronously. A slow handler looks like a failed delivery and triggers redundant retries. Second, when you call the provider (not just receive from it), apply the same discipline in reverse — retry with backoff and jitter, and carry the same idempotency key on every attempt so a retried charge never becomes two charges.

What ordering guarantees do payment events give you?

None. Assume events can arrive in any order and design so it never matters. Stripe: “Stripe doesn’t guarantee the delivery of events in the order that they’re generated,” and warns that a single action can emit several events that appear in any sequence. The correct response is not to reconstruct order but to stop depending on it.

There are two reliable tools. The state machine above absorbs out-of-order events by rejecting stale transitions. And when you genuinely need current truth, fetch it: “use the API to retrieve any missing objects.” The webhook is a notification that something changed — a hint to go look — not the authoritative record of what the value now is. The provider’s API is your source of truth; the webhook only tells you when to consult it. A periodic reconciliation job that re-fetches recent objects and repairs any drift closes the gap for events that were dropped entirely. Building that reconciliation muscle is part of a durable fintech stack for 2026.

What is the outbox pattern and why do you need it?

The outbox pattern guarantees that you emit an event if and only if the database change it describes committed. You write the domain change and an “outbox” row in the same local transaction, then a separate relay reads the outbox and publishes the events. Because both writes share one transaction, you can never end up having charged a customer but failed to emit the event, or emitted an event for a change that rolled back — the dual-write problem that silently corrupts ledgers.

Failure mode	Symptom	Mitigation
Duplicate delivery	Same event id processed twice; double side effect	Dedupe on event id; idempotent upsert handler
Out-of-order arrival	Stale event overwrites newer state	Per-entity state machine that ignores backward transitions
Forged or replayed event	Fake webhook triggers fulfillment or access	HMAC verification plus timestamp tolerance window
Dual-write gap	DB change committed but event never emitted	Transactional outbox with a separate relay/publisher
Dropped delivery	Event never arrives; state silently diverges	Scheduled reconciliation against the provider API

The relay itself is at-least-once — it may publish a row twice if it crashes after sending but before marking it done — which is exactly why the consumer side must already be idempotent. The two patterns are complementary: the outbox guarantees emission, and consumer-side dedup absorbs the duplicates emission can produce. Whether to build this in-house or lean on managed infrastructure is its own decision, covered in build vs. buy fintech infrastructure.

Getting the plumbing right

None of this is exotic. It is a small set of patterns — client keys, event-id dedup, state machines, signature-plus-timestamp verification, backoff, and the outbox — applied without exception across every money-moving path. The failures come from applying them in four places out of five and assuming the fifth will be fine.

FinWeb builds and audits exactly this layer as part of our platform engineering work: idempotent handlers, verified webhooks, and reconciliation that proves your ledger matches the provider’s. If you want a second set of eyes on your payment event pipeline before it ships, talk to us.