Idempotency Keys: How to Make Retries Safe in Distributed Systems

A customer clicks "Pay Now" once. The request reaches your payment service, the charge succeeds, but the response times out on the way back. The client, seeing no answer, retries. Somewhere in your logs there are now two successful charges for one click. Nobody wrote a bug. The network just did what networks do.

Retries Are Not Optional, They're the Default

In a single process, a function call either returns or the whole program crashes with it. In a distributed system, that guarantee disappears. A request can fail before it reaches the server, after the server processes it but before the response comes back, or anywhere in between, and from the caller's side, all three failures look identical: silence, or a timeout.

Given that, a caller has exactly two honest choices: give up, or retry. Almost every serious system chooses to retry, because giving up on a payment, an order, or a blockchain transaction just because a router hiccupped is worse than the alternative. The problem is that "the alternative" retrying turns a network problem into a correctness problem, unless the operation you're retrying can tolerate being run more than once.

That property has a name: idempotency. An operation is idempotent if running it once and running it five times leave the system in the same state. SET x = 5 is idempotent — running it twice still leaves x at 5. x = x + 5 is not, run it twice and you've added 10. Retrying a network call is safe by default only when the operation behind it is naturally idempotent. Charging a card, placing an order, sending an email — none of these are. That's the gap idempotency keys exist to close.

How an Idempotency Key Actually Works

The mechanism is simpler than the name suggests. The client generates a unique key, typically a UUID, once, at the moment the user takes the action, before any request is sent. That key is attached to the request, usually as a header (Idempotency-Key: 8f14e...). Every retry of that same logical action, the same click, the same order, reuses the exact same key.

On the server, before doing any real work, the first thing that happens is a lookup: has this key been seen before?

• Not seen before → this is genuinely a new request. Process it (charge the card, place the order), store the key alongside the result, and return.
• Seen before, and finished → this is a retry of something already done. Skip the actual work entirely and return the stored result from the first attempt. The client gets the same success response it would have gotten the first time, and no charge happens twice.
• Seen before, and still in progress → a retry arrived while the original request is still being processed (a slow response, not a failed one). The correct move is to make the retry wait or reject it, never to run the operation again concurrently.

That storage of "key → result" is the entire trick. It turns "run this action" into "run this action, but only the first time you're asked, and remember what happened so every later ask gets the same answer." The database row or cache entry holding that key is doing the same job as the WAL in PostgreSQL, giving you a durable record of intent so a retry, a crash, or a redelivery doesn't have to guess what already happened.

Why This Matters in Production, Not Just in Theory

This isn't a hypothetical edge case; it's the default behavior of almost every reliability mechanism you already rely on:

• Payment gateways retry. Stripe, for instance, builds idempotency keys into its API specifically because a client-side timeout on a successful charge is common enough to have a name in their docs.
• Message queues retry. SQS, Kafka consumers, and most queue systems offer at-least-once delivery, not exactly-once, because exactly-once delivery across a network is provably expensive to guarantee. "At least once, so dedupe on your end" is the honest tradeoff, and idempotency keys are "your end." This is exactly why background job systems like BullMQ build retries with exponential backoff and dead letter queues into the framework itself, rather than leaving it to each job handler to reinvent.
• Mobile clients retry. A user on a bad connection taps "Submit" once, but the app, seeing no response after a few seconds, quietly retries the request behind the scenes. Without an idempotency key, that one tap becomes two orders.
• Load balancers and proxies retry. Some infrastructure automatically retries a request that failed to connect, before your application code ever sees it happen once, let alone twice.
• Webhooks retry, by design. Stripe, GitHub, and most webhook senders will redeliver an event if your endpoint doesn't return a fast 2xx response, on the assumption that a timeout means you never got it. Idempotent webhook processing — checking the event ID before acting on it — is the only thing standing between that redelivery and a duplicate side effect on your end.

Without an idempotency key sitting between "the network is unreliable" and "the operation isn't naturally repeatable," every one of these standard, unremarkable mechanisms becomes a live double-charge, double-order, or double-send bug waiting for the right timing to trigger it.

Example: Why a Blockchain Indexer Cannot Survive Without This

Nowhere is this more visible, or more unforgiving, than in a blockchain indexer, and it's worth walking through concretely.

An indexer's whole job is to watch a chain, pull each new block, extract the events or transfers inside it, and write them into your own database so your application can query them fast. That sounds like a simple one-pass pipeline: get block, process block, move on. In practice it never runs exactly once per block, for reasons entirely outside your control:

• Reorgs. The chain itself can rewind and replay a range of blocks when a fork resolves. Your indexer will see block 18,402,001 more than once, as two different, competing versions of "reality."
• Crash recovery. If your indexer process dies mid-batch, on restart it doesn't know precisely which of the last few blocks were fully written versus half-written, so the safe move is to reprocess a small overlapping window, not trust a fragile "last processed block" pointer to be exactly right.
• RPC retries. The call to fetch a block from a node can itself time out and get retried, occasionally landing you the same block payload twice from two separate requests racing each other.

Now picture what happens without idempotency built in: a transfer event says "500 USDC moved from wallet A to wallet B." If that event gets processed twice, because of a reorg replay or a crash-recovery reprocess, and your indexing logic does balance += 500 each time it sees the event, wallet B's balance is now wrong by 500 USDC in your database, permanently, until someone notices the number doesn't match the chain and re-runs a full backfill. On a system indexing tens of millions of events, that's not a rare accident, it's a near-certainty over enough uptime.

The fix is the exact same pattern as the payment example, just with a different key. Instead of a client-generated UUID, the natural idempotency key is something already unique to the data itself: (transaction_hash, log_index) for an EVM chain, for instance, uniquely identifies one specific event, no matter how many times it's delivered to you. The write becomes an upsert keyed on that pair, not a blind balance += amount. Seen this (tx_hash, log_index) before? Skip it, or overwrite with the identical result, never add on top of it again. Reorgs, crash-recovery reprocessing, and duplicate RPC responses all become harmless, because the operation of "record this event" is now idempotent, exactly the same property a payment API gets from a client-supplied key.

Where to Put the Key

Idempotency keys work at whatever layer needs the guarantee, and the source of the key changes depending on who's best positioned to know "this is the same logical request":

• Client-generated (payments, order placement, form submissions): the client mints a UUID once, before the first attempt, and resends the identical key on every retry of that same user action.
• Data-derived (blockchain events, webhook deliveries, message queue consumers): the key comes from something already unique in the payload itself, a transaction hash, a webhook delivery ID, a message ID, so you never have to coordinate key generation between sender and receiver at all.

Either way, the underlying requirement is the same: the key must be stored durably, checked before any side effect runs, and scoped to an appropriate time window (payment idempotency keys are typically only guaranteed for 24 hours; a blockchain indexer's (tx_hash, log_index) uniqueness is effectively permanent).

Where This Fits in the Full Course

Idempotent webhook processing and BullMQ retry/backoff patterns are both covered hands-on in the Node.js In-Depth course, in the Practitioner phase — module P-7 ("Connecting External Services and Caching") and module P-12 ("Background Jobs and Task Queues with BullMQ"), respectively.

The Question Worth Asking Yourself

Next time you write a piece of code that has a side effect, charges something, sends something, increments something, and sits behind a network call, ask one question before you ship it: if this exact request arrived twice, on purpose or by accident, would the result still be correct?

If the honest answer is no, that's not a rare failure mode you're accepting, it's a bug you've already written. The network will find it for you eventually. It's better to find it first.

#DistributedSystems #SoftwareEngineering #Backend #SystemDesign #Blockchain #APIDesign